Ubuntu 18.04 x64 build failing #27954

sdmaclea · 2018-11-19T17:54:21Z

On the current tip, I am seeing build failures on Linux x64.

The ./build.sh script output ends with a message

Build failed (exit code '139'). See log: /home/stmaclea/git/corefx/eng/common/../../artifacts/log/Debug/Build.binlog

The Build.binlog is a binary file and does not seem to contain useful information.

Early in the console output there are a series of warnings

~/git/corefx$ ./build.sh 
Downloading 'https://dot.net/v1/dotnet-install.sh'
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-sdk-2.1.401-linux-x64.tar.gz
dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-sdk-2.1.401-linux-x64.tar.gz
dotnet-install: Adding to current process PATH: `/home/stmaclea/git/corefx/.dotnet`. Note: This change will be visible only when sourcing script.
dotnet-install: Installation finished successfully.
/home/stmaclea/git/corefx/.dotnet/sdk/2.1.401/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/home/stmaclea/git/corefx/eng/common/../../artifacts/log/Debug/ToolsetRestore.binlog /clp:Summary /clp:None /nr:true /p:__ToolsetLocationOutputFile=/home/stmaclea/git/corefx/eng/common/../../artifacts/toolset/1.0.0-beta.18565.8.txt /t:__WriteToolsetLocation /home/stmaclea/git/corefx/eng/common/../../artifacts/toolset/restore.proj

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.23

/home/stmaclea/git/corefx/.dotnet/sdk/2.1.401/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/home/stmaclea/git/corefx/eng/common/../../artifacts/log/Debug/Build.binlog /clp:Summary /nr:true /p:Configuration=Debug /p:Projects=/home/stmaclea/git/corefx/eng/common/../../*.sln /p:RepoRoot=/home/stmaclea/git/corefx/eng/common/../.. /p:Restore=true /p:Build=true /p:Rebuild=false /p:Test=false /p:Pack=false /p:IntegrationTest=false /p:PerformanceTest=false /p:Sign=false /p:Publish=false /p:ContinuousIntegrationBuild=false /home/stmaclea/.nuget/packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18565.8/tools/Build.proj
  Restoring packages for /home/stmaclea/.nuget/packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18565.8/tools/Tools.proj...
  Generating MSBuild file /home/stmaclea/git/corefx/artifacts/toolset/Common/Tools.proj.nuget.g.props.
  Generating MSBuild file /home/stmaclea/git/corefx/artifacts/toolset/Common/Tools.proj.nuget.g.targets.
  Restore completed in 231.43 ms for /home/stmaclea/.nuget/packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18565.8/tools/Tools.proj.
  Restoring packages for /home/stmaclea/git/corefx/external/netstandard/netstandard.depproj...
  Generating MSBuild file /home/stmaclea/git/corefx/artifacts/obj/netstandard/netstandard/netstandard.depproj.nuget.g.props.
  Generating MSBuild file /home/stmaclea/git/corefx/artifacts/obj/netstandard/netstandard/netstandard.depproj.nuget.g.targets.
  Restore completed in 31.45 ms for /home/stmaclea/git/corefx/external/netstandard/netstandard.depproj.
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning : The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.NativeMethods.git_buf_free(GitBuf buf) [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.Proxy.ConvertPath(Func`2 pathRetriever) [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Repository.Discover(String startingPath) [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.GitOperations.LocateRepository(String directory) in /_/src/Microsoft.Build.Tasks.Git.Operations/GitOperations.cs:line 26 [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.RepositoryTasks.LocateRepository(LocateRepository task) in /_/src/Microsoft.Build.Tasks.Git.Operations/RepositoryTasks.cs:line 58 [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]
/home/stmaclea/.nuget/packages/microsoft.sourcelink.common/1.0.0-beta-63401-01/build/Microsoft.SourceLink.Common.targets(53,5): warning : Source control information is not available - the generated source link is empty. [/home/stmaclea/git/corefx/external/netstandard/netstandard.depproj]

The text was updated successfully, but these errors were encountered:

sdmaclea · 2018-11-19T17:55:33Z

I have tried deleting my ~/.nuget directory and tried git clean -xdf. I have also cleaned /tmp of dotnet related stuff. I also revisited the corefx unix build instructions to make sure all dependencies were installed.

karelz · 2018-11-19T19:03:50Z

@sdmaclea do you get same problems on clean Linux VM? The builds are fine in CI, so I wonder if you have some tainted environment.

cc @wtgodbe

sdmaclea · 2018-11-19T19:09:51Z

I am also running Ubuntu 18.04 if that is relevant.

tannergooding · 2018-11-19T19:19:05Z

The Build.binlog is a binary file and does not seem to contain useful information.

You need the binary log viewer: https://github.com/Microsoft/msbuild/blob/master/documentation/wiki/Binary-Log.md

sdmaclea · 2018-11-19T20:56:35Z

I just tried with my azure Linux x64 Ubuntu 18.04 VM as well. I get similar symptoms.

I'll try a fresh 18.04 VM.

jkotas · 2018-11-19T21:12:09Z

Build failed (exit code '139')

This is hard crash. Is it deterministic or does it go away when you retry the build?

We seem to be hitting bad crashes during the build more frequently recently (another instance: https://github.com/dotnet/coreclr/issues/20487). Maybe related to arcade tooling updates?

sdmaclea · 2018-11-19T21:19:33Z

It seems to be repeatable. The log seems to end about the same place each time.

jkotas · 2018-11-19T21:28:40Z

This crash should generate coredump. Could you please try to get the coredump and open it in lldb to see where it is crashing?

sdmaclea · 2018-11-19T22:00:30Z

This is the call stack

* thread dotnet/corefx#1, name = 'dotnet', stop reason = signal SIGSEGV
  * frame #0: 0x00007faf6951cac8 ld-linux-x86-64.so.2`___lldb_unnamed_symbol42$$ld-linux-x86-64.so.2 + 888
    frame dotnet/corefx#1: 0x00007faf695250bd ld-linux-x86-64.so.2`___lldb_unnamed_symbol81$$ld-linux-x86-64.so.2 + 765
    frame dotnet/corefx#2: 0x00007faf685242df libc.so.6`__GI__dl_catch_exception(exception=0x00007faeb4bb4120, operate=(ld-linux-x86-64.so.2`___lldb_unnamed_symbol81$$ld-linux-x86-64.so.2), args=0x00007faeb4bb4140) at dl-error-skeleton.c:196
    frame dotnet/corefx#3: 0x00007faf695247ca ld-linux-x86-64.so.2`___lldb_unnamed_symbol79$$ld-linux-x86-64.so.2 + 186
    frame dotnet/corefx#4: 0x00007faf6930cf96 libdl.so.2`dlopen_doit(a=0x00007faeb4bb4370) at dlopen.c:66
    frame dotnet/corefx#5: 0x00007faf685242df libc.so.6`__GI__dl_catch_exception(exception=0x00007faeb4bb4310, operate=(libdl.so.2`dlopen_doit at dlopen.c:58), args=0x00007faeb4bb4370) at dl-error-skeleton.c:196
    frame dotnet/corefx#6: 0x00007faf6852436f libc.so.6`__GI__dl_catch_error(objname=0x00007fae9403ffd0, errstring=0x00007fae9403ffd8, mallocedp=0x00007fae9403ffc8, operate=<unavailable>, args=<unavailable>) at dl-error-skeleton.c:215
    frame dotnet/corefx#7: 0x00007faf6930d735 libdl.so.2`_dlerror_run(operate=(libdl.so.2`dlopen_doit at dlopen.c:58), args=0x00007faeb4bb4370) at dlerror.c:162
    frame dotnet/corefx#8: 0x00007faf6930d051 libdl.so.2`__dlopen at dlopen.c:87
    frame dotnet/corefx#9: 0x00007faf6930d030 libdl.so.2`__dlopen(file=<unavailable>, mode=<unavailable>)
    frame dotnet/corefx#10: 0x00007faf67b0702a libcoreclr.so`LoadLibraryExW + 474
    frame dotnet/corefx#11: 0x00007faf676fced7 libcoreclr.so`LoadLibraryExWrapper(char16_t const*, void*, unsigned int) + 359
    frame dotnet/corefx#12: 0x00007faf677affe6 libcoreclr.so`CLRLoadLibrary(char16_t const*) + 38
    frame dotnet/corefx#13: 0x00007faf6790da64 libcoreclr.so`NDirect::LoadLibraryModule(NDirectMethodDesc*, LoadLibErrorTracker*) + 1284
    frame dotnet/corefx#14: 0x00007faf6790c88f libcoreclr.so`NDirect::NDirectLink(NDirectMethodDesc*) + 175
    frame dotnet/corefx#15: 0x00007faf6790c7b6 libcoreclr.so`NDirect::GetStubForILStub(NDirectMethodDesc*, MethodDesc**, unsigned int) + 246
    frame dotnet/corefx#16: 0x00007faf6790ccd7 libcoreclr.so`GetStubForInteropMethod(MethodDesc*, unsigned int, MethodDesc**) + 375
    frame dotnet/runtime#13850: 0x00007faf67929ee0 libcoreclr.so`MethodDesc::DoPrestub(MethodTable*) + 1136
    frame dotnet/runtime#13851: 0x00007faf679297f4 libcoreclr.so`PreStubWorker + 452
    frame dotnet/corefx#19: 0x00007faf678abb94 libcoreclr.so`ThePreStub + 92
    frame dotnet/runtime#13852: 0x00007faef071a939
    frame dotnet/corefx#21: 0x00007faef074a887
    frame dotnet/runtime#13853: 0x00007faef07073f3
    frame dotnet/corefx#23: 0x00007faef070896d
    frame dotnet/corefx#24: 0x00007faef070871b
    frame dotnet/corefx#25: 0x00007faef074fbd9
    frame dotnet/runtime#13854: 0x00007faeee17d63d
    frame dotnet/corefx#27: 0x00007faef26f7970
    frame dotnet/corefx#28: 0x00007faeee20d338
    frame dotnet/corefx#29: 0x00007faf678aaeaf libcoreclr.so`CallDescrWorkerInternal + 124
    frame dotnet/corefx#30: 0x00007faf677cb5dc libcoreclr.so`MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 940
    frame dotnet/corefx#31: 0x00007faf6794b865 libcoreclr.so`QueueUserWorkItemManagedCallback(void*) + 181
    frame dotnet/corefx#32: 0x00007faf6779cf0d libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 413
    frame dotnet/corefx#33: 0x00007faf6779d670 libcoreclr.so`ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 64
    frame dotnet/corefx#34: 0x00007faf6792fa67 libcoreclr.so`ManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 295
    frame dotnet/runtime#13855: 0x00007faf677bbd03 libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) + 1267
    frame dotnet/runtime#13856: 0x00007faf67b31a62 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 306
    frame dotnet/corefx#37: 0x00007faf690f46db libpthread.so.0`start_thread + 219
    frame dotnet/corefx#38: 0x00007faf684de88f libc.so.6`__GI___clone at clone.S:95

jkotas · 2018-11-19T22:25:49Z

This looks like a crash while trying to load unmanaged library. Could you please try to find the name of the library that this is trying to load?

sdmaclea · 2018-11-19T23:37:02Z

I tried a clean Ubuntu 18.04 VM, it had the same issue.

sdmaclea · 2018-11-20T00:40:33Z

Looks like it is trying to load libc.so

Strace fragment

3287  openat(AT_FDCWD, "/home/stmaclea/git/corefx/.dotnet/shared/Microsoft.NETCore.App/2.1.3/libc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or direct
ory)
3287  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 121
3287  access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libc.so", O_RDONLY|O_CLOEXEC) = 121
3287  openat(AT_FDCWD, "/home/stmaclea/git/corefx/.dotnet/shared/Microsoft.NETCore.App/2.1.3/liblibc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or dir
ectory)
3287  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 121
3287  access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblibc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/liblibc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/lib/liblibc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/usr/lib/liblibc.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3287  openat(AT_FDCWD, "/home/stmaclea/git/corefx/.dotnet/shared/Microsoft.NETCore.App/2.1.3/libc", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory
)
3287  --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x38} ---
3287  unlink("/tmp/clr-debug-pipe-3166-416831-in") = 0
3287  unlink("/tmp/clr-debug-pipe-3166-416831-out") = 0
3287  --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x38} ---

Strange thing is it found libc, opened it and kept looking... Perhaps not the expected version?

sdmaclea · 2018-11-20T01:04:42Z

In Ubuntu 18.04, libc.so contains

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libc.so.6 /usr/lib/x86_64-linux-gnu/libc_nonshared.a  AS_NEEDED ( /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 ) )

sdmaclea · 2018-11-20T16:29:44Z

I created 14.04 & 16.04 VMs. Both build corefx successfully.

ViktorHofer · 2018-12-10T18:19:07Z

Also failing for me consistently on Ubuntu 18.04.

/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning : The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.NativeMethods.git_buf_free(GitBuf buf) [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.Proxy.ConvertPath(Func`2 pathRetriever) [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Repository.Discover(String startingPath) [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.GitOperations.LocateRepository(String directory) in /_/src/Microsoft.Build.Tasks.Git.Operations/GitOperations.cs:line 26 [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.RepositoryTasks.LocateRepository(LocateRepository task) in /_/src/Microsoft.Build.Tasks.Git.Operations/RepositoryTasks.cs:line 58 [/home/vihofer/git/corefx/external/runtime/runtime.depproj]
/home/vihofer/.nuget/packages/microsoft.sourcelink.common/1.0.0-beta-63401-01/build/Microsoft.SourceLink.Common.targets(53,5): warning : Source control information is not available - the generated source link is empty. [/home/vihofer/git/corefx/external/runtime/runtime.depproj]

danmoseley · 2018-12-11T05:22:45Z

dotnet/arcade#1261

danmoseley · 2018-12-11T05:23:25Z

https://github.com/dotnet/corefx/issues/33862

MarcoRossignoli · 2018-12-11T10:20:55Z

Same issue(my track comment).

danmoseley · 2018-12-11T21:15:01Z

dotnet/sourcelink#197

danmoseley · 2018-12-11T21:15:47Z

libgit2/libgit2sharp#1583
GitTools/GitVersion#1497

danmoseley · 2018-12-11T21:18:42Z

Seems this is an issue of fragility in loading dependencies of LibGit2Sharp which is used by Microsoft.Build.Tasks.Git.RepositoryTasks which is owned by SourceLink:
https://github.com/dotnet/sourcelink/blob/c092238370e0437eb95722f28c79273244dc7f1a/src/Microsoft.Build.Tasks.Git.Operations/RepositoryTasks.cs

omajid · 2018-12-14T21:59:04Z

FWIW, I am getting this error when running ./build.sh on a fresh Fedora 25 machine too.

/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning : The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. [/home/ci/corefx/src/Native/build-native.proj]
/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.NativeMethods.git_buf_free(GitBuf buf) [/home/ci/corefx/src/Native/build-native.proj]       
/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Core.Proxy.ConvertPath(Func`2 pathRetriever) [/home/ci/corefx/src/Native/build-native.proj]      
/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at LibGit2Sharp.Repository.Discover(String startingPath) [/home/ci/corefx/src/Native/build-native.proj]
/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.GitOperations.LocateRepository(String directory) in /_/src/Microsoft.Build.Tasks.Git.Operations/GitOperations.cs:line 26 [/home/ci/corefx/src/Native/build-native.proj]                                       
/home/ci/.nuget/packages/microsoft.build.tasks.git/1.0.0-beta-63401-01/build/Microsoft.Build.Tasks.Git.targets(20,5): warning :    at Microsoft.Build.Tasks.Git.RepositoryTasks.LocateRepository(LocateRepository task) in /_/src/Microsoft.Build.Tasks.Git.Operations/RepositoryTasks.cs:line 58 [/home/ci/corefx/src/Native/build-native.proj]                             
/home/ci/.nuget/packages/microsoft.dotnet.arcade.sdk/1.0.0-beta.18613.4/tools/Version.targets(69,5): error : SourceRevisionId is not set, which means the SourceLink targets are not included in the build. Those are needed to produce a correct sha for our build outputs. [/home/ci/corefx/src/Native/build-native.proj]

danmoseley · 2018-12-15T06:14:06Z

Please note that in the source link issue.

wtgodbe · 2018-12-20T20:58:35Z

@danmosemsft so is this https://github.com/dotnet/corefx/issues/33987, and was therefore fixed with dotnet/corefx#34186?

ViktorHofer · 2018-12-20T21:03:11Z

No that's something different. See the linked issue

ViktorHofer · 2019-01-05T10:57:20Z

Fixed with dotnet/arcade@bf5f600

sdmaclea changed the title ~~Linux x64 build failing~~ Ubuntu 18.04 x64 build failing Nov 20, 2018

ViktorHofer closed this as completed Jan 5, 2019

msftgits transferred this issue from dotnet/corefx Jan 31, 2020

msftgits added this to the 3.0 milestone Jan 31, 2020

ghost locked as resolved and limited conversation to collaborators Dec 15, 2020

Ubuntu 18.04 x64 build failing #27954

Ubuntu 18.04 x64 build failing #27954

Comments

sdmaclea commented Nov 19, 2018

sdmaclea commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karelz commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdmaclea commented Nov 19, 2018

Uh oh!

tannergooding commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdmaclea commented Nov 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkotas commented Nov 19, 2018

Uh oh!

sdmaclea commented Nov 19, 2018

Uh oh!

jkotas commented Nov 19, 2018

Uh oh!

sdmaclea commented Nov 19, 2018

Uh oh!

jkotas commented Nov 19, 2018

Uh oh!

sdmaclea commented Nov 19, 2018

Uh oh!

sdmaclea commented Nov 20, 2018

Uh oh!

sdmaclea commented Nov 20, 2018

Uh oh!

sdmaclea commented Nov 20, 2018

Uh oh!

ViktorHofer commented Dec 10, 2018

Uh oh!

danmoseley commented Dec 11, 2018

Uh oh!

danmoseley commented Dec 11, 2018

Uh oh!

MarcoRossignoli commented Dec 11, 2018

Uh oh!

danmoseley commented Dec 11, 2018

Uh oh!

danmoseley commented Dec 11, 2018

Uh oh!

danmoseley commented Dec 11, 2018

Uh oh!

omajid commented Dec 14, 2018

Uh oh!

danmoseley commented Dec 15, 2018

Uh oh!

wtgodbe commented Dec 20, 2018

Uh oh!

ViktorHofer commented Dec 20, 2018

Uh oh!

ViktorHofer commented Jan 5, 2019

Uh oh!

sdmaclea commented Nov 19, 2018 •

edited

Loading

karelz commented Nov 19, 2018 •

edited

Loading

tannergooding commented Nov 19, 2018 •

edited

Loading

sdmaclea commented Nov 19, 2018 •

edited

Loading