Sentry missing our source code on iOS

[gnprice]

We've been seeing errors like this one on some Sentry events (example):

That's "Source code was not found". The consequence is that the stack trace is pretty cryptic.

That event was on iOS, on release 26.20.143 which is the latest. I built that release with the usual tools/ios build command, which exited with success. It sounds like the Sentry code in the build either didn't run, or didn't succeed in uploading the source code to the Sentry service.

[gnprice]

(This may be easiest to debug next week, when I'm in the office and we can examine the log file left behind from that build on the Mac where I made the build; right now I'm traveling and don't have access to it.)

[gnprice]

Well, this was pretty straightforward once I looked at it. Here's from the log:

PhaseScriptExecution Upload\ Debug\ Symbols\ to\ Sentry /Users/greg/Library/Developer/Xcode/
DerivedData/ZulipMobile-diuocloqwafvdpeozujwphdrhalf/Build/Intermediates.noindex/ArchiveInte
rmediates/ZulipMobile/IntermediateBuildFilesPath/ZulipMobile.build/Release-iphoneos/ZulipMob
ile.build/Script-6FD412E618344A7FB938E4AC.sh (in target 'ZulipMobile' from project 'ZulipMob
ile')
[...]
    /bin/sh -c /Users/greg/Library/Developer/Xcode/DerivedData/ZulipMobile-diuocloqwafvdpeoz
ujwphdrhalf/Build/Intermediates.noindex/ArchiveIntermediates/ZulipMobile/IntermediateBuildFi
lesPath/ZulipMobile.build/Release-iphoneos/ZulipMobile.build/Script-6FD412E618344A7FB938E4AC
.sh
[Upload Debug Symbols to Sentry] Skipping sentry

"Skipping sentry". Why would it do that? Well, here's the logic that controls that choice:

			shellScript = "if [[ $CONFIGURATION == \"Release\" && ! `git describe --tags` =~ '-g' ]]; then\n  export SENTRY_PROPERTIES=sentry.properties\n  ../node_modules/@sentry/cli/bin/sentry-cli upload-dsym\nelse\n  echo [Upload Debug Symbols to Sentry] Skipping sentry\nfi\n";

i.e.

if [[ $CONFIGURATION == "Release" && ! `git describe --tags` =~ '-g' ]]; then
  export SENTRY_PROPERTIES=sentry.properties
  ../node_modules/@sentry/cli/bin/sentry-cli upload-dsym
else
  echo [Upload Debug Symbols to Sentry] Skipping sentry
fi

This will never work, because in a real release build to be published, HEAD is not a tagged commit but instead has a small commit on top of the tagged commit, adding Sentry secrets to the tree. (A line each in android/sentry.properties, ios/sentry.properties, and src/config.js.) In particular this is how the auth token for making an upload to Sentry gets filled into the sentry.properties file mentioned in that scriptlet -- so that step can never actually work at the commits we tag.

This logic was added in b4c11e0 / #3676, replacing some logic that worked correctly for real releases and for the most common debugging/development cases but could be over-inclusive in other cases.

A quick fix would be to simply revert that commit. I may do this for right now to just get things working again.

For a more precise fix, I think what I want is to make the particular build explicitly request Sentry -- just like we do on Android. For Xcode I think this would look like adding a build setting -- which for these scripts becomes an environment variable -- and then we'd set that on the xcodebuild command line in tools/ios, like USE_SENTRY=YES.

[gnprice]

A quick fix would be to simply revert that commit. I may do this for right now to just get things working again.

I tried this locally, and went and rebuilt 26.28.151 with just that change, and the "build number" (CFBundleVersion) bumped.

It worked as far as getting the files uploaded to Sentry! Here's what the artifacts page for that release now looks like:

showing ~/main.jsbundle and ~/main.jsbundle.map. (The ~/ prefix seems like a malapropism but there it is.)

Compare to the one for the previous release, showing no artifacts:

And spot-checking older releases, there seem to be no artifacts going back to 26.16.139, the first after that change b4c11e0. Whereas just before that, it looks very much like 26.28.151 now looks (complete with the funny ~/ prefix):

OTOH we're still not getting good stack traces for events on 26.28.151 on iOS -- even on a new event I provoked after that upload. ... Ah, I think it's because I bumped the build number. Good for Sentry, then -- it's right to be making that distinction. I'll rerun the build, with Sentry included, back on the old build number and we'll see if that gets us stack traces for the already-in-beta release.

[gnprice]

I'll rerun the build, with Sentry included, back on the old build number and we'll see if that gets us stack traces for the already-in-beta release.

And this worked! Huzzah. Here's an event with real source code on the JS stack trace:
https://sentry.io/organizations/zulip/issues/1660211451/events/8f3054b62c5043d2a2dd1877ca07e905/

Compare to where I provoked the exact same error before that last upload:
https://sentry.io/organizations/zulip/issues/1660172423/events/a665582a72914438923ae8485153adb5/

So I think we should be seeing source code on our stack traces for the 26.28.151 release (the current beta) from here on.

I'll also make that revert on master so that we get it automatically again on future releases. And I'll open a followup issue to clean up that logic better, as I outlined above at the end of #3809 (comment) .