Add OpenSSL::SSL::Context::Server#on_server_name for SNI

[carlhoerberg]

Summary

• Adds on_server_name method to OpenSSL::SSL::Context::Server for Server Name Indication (SNI) support
• Allows TLS servers to present different certificates based on the client's requested hostname
• Enables virtual hosting over TLS with a single listening socket

Example

default_context = OpenSSL::SSL::Context::Server.new
default_context.certificate_chain = "default.crt"
default_context.private_key = "default.key"

example_context = OpenSSL::SSL::Context::Server.new
example_context.certificate_chain = "example.com.crt"
example_context.private_key = "example.com.key"

default_context.on_server_name do |hostname|
  case hostname
  when "example.com", "www.example.com"
    example_context
  else
    nil # use default context
  end
end

Test plan

• Added spec for on_server_name method in context_spec.cr
• Added end-to-end spec in socket_spec.cr that verifies the callback receives the client hostname

🤖 Generated with Claude Code

[straight-shoota]

Apparently this is broken in the interpreter. Perhaps we could just skip the spec in interpreted mode? With a note that this leads to invalid memory access.

And Crystal 1.0 doesn't seem to support the proc literal syntax. Might be best to use Proc.new instead: Proc(LibSSL::SSL, LibC::Int*, Void*, LibC::Int).new { |ssl, alert_ptr, arg|

[ysbaddaden]

Just a polish change to make sure the c callback is properly validated by the compiler (no closure data) when sent to the OpenSSL function.

[straight-shoota]

I have some ideas for improving this feature: #16494

[carlhoerberg]

We've started to notice GC Warning: Finalization cycle involving (???) when running SNI specs. We suspect it's related to the storage of the callback, @sni_callback_box. One way we can avoid it is to use a class level hash for storing the callback box, and deleting it in the finalizer, but that seems less than optimal..

[ysbaddaden]

I assume the issue comes from the captured block closure: it can have direct references to other contexts and trying to finalize one context leads the GC to identify a circular finalization... which isn't an issue because the finalizer only accesses the object itself.

I wish there was a way to tell the GC to skip the check (unsafe finalizer).

[carlhoerberg]

I wish there was a way to tell the GC to skip the check (unsafe finalizer).

how about GC_register_finalizer_ignore_self? https://github.com/bdwgc/bdwgc/blob/master/docs/finalization.md#getting-around-topological-finalization-ordering

[straight-shoota]

We already use GC_register_finalizer_ignore_self for all finalizers:

crystal/src/gc/boehm.cr

Lines 338 to 343 in 37cdfa7

	private def self.add_finalizer_impl(object : T) forall T
	LibGC.register_finalizer_ignore_self(object.as(Void*),
	->(obj, data) { obj.as(T).finalize },
	nil, nil, nil)
	nil
	end