Release built docker image with unicode encoder issue

[jan-johansson-mr]

I've opened an issue that I thought maybe is better suited in this area.

What the issue is

The Swedish letters å, ä, ö, Å, Ä and Ö are not properly encoded in a release built Linux container image, based on aspnetcore runtime.

How to reproduce

Simply create a web app from VS 2022 template, remove all the setup code and so on, and leave one liner

Console.WriteLine("åäöÅÄÖ");

Then release build the Linux container image (e.g. docker) and run it. The letters are printed as

U+FFFD U+FFFD U+FFFD U+FFFD U+FFFD U+FFFD (I've inserted spaces for readability)

What was the expected behavior and how to reproduce that

Create a console app from VS 2022 template, and leave the one liner as

Console.WriteLine("åäöÅÄÖ");

Then release build the Linux container image (e.g. docker) and run it. The letters are printed as

åäöÅÄÖ

How to resolve the issue with the web app

I have no idea. I've tried different approaches as I've described in the original thread, but nothing works.

What is effected

I saw the issue first with my web app, and started the investigation. I have no resolution to the issue.

[jan-johansson-mr]

Here is a snapshot of my environment (dotnet --info)

.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  5.0.401 [C:\Program Files\dotnet\sdk]
  6.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 5.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[pranavkm]

Are your files stored with UTF8 encoding / UTF8+BOM encoding? We've seen other issues where files are stored using Window's CodePage encoding which results in garbled text on non-Windows OSes.

[ghost]

Hi @jan-johansson-mr. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[jan-johansson-mr]

Hi @pranavkm,

This was very interesting! :-)

The web app is stored with Window's code page Windows-1252 (CP1252) character set, while the console app is stored with UTF8+BOM encoding. This might actually be the source of the issue I see. It explains why there is no problem with the console app in my investigation, but there is a problem with the web app.

It also tells me that the template for web app in VS 2022 comes with CP1252, while other template (e.g. console app) comes with UTF8+BOM.

I saved the file (Program.cs) in the web app with Unicode (UTF-8 with signature) - Codepage 65001 (UTF8+BOM) and now the characters are displayed fine in the web app too (Linux container image), as it is in console app.

This was tricky! But I'm happy :-)
Thanks!

Now I'm thinking that there has to be some VS 2022 settings somewhere that tells VS to save all my files (including new ones) as UTF8+BOM. I hope there is one...

[pranavkm]

that there has to be some VS 2022 settings somewhere that tells VS to save all my files (including new ones) as UTF8+BOM. I hope there is one...

I've found using an .editorconfig works really well for this:

[*.cs]
charset = utf-8-bom # or utf-8

We're using #37201 to ensure the files .NET Core project or item templates scaffold use a BOM since it makes things a lot more predictable.

[jan-johansson-mr]

Thanks, closing this issue since it's handled by another thread.