LoadCertificateAndKey, an internal method of System.Net.HttpListener, accepts the IP address and port of the binding end point as arguments, but does nothing with the address. Instead, it loads the certificate from the port number alone, forcing all addresses to be bound to the same certificate and private key.
Ideally LoadCertificateAndKey would try to load by address and port, than fall back to loading by port. This would provide backwards compatibility with existing setups while allowing new implementations to bind different certificates to different IP addresses.
If a pull request would help, I'd be happy to submit one.
https://github.com/mono/mono/blob/master/mcs/class/System/System.Net/HttpListener.cs#L90 (link valid as of b48cd9b3c1fff64fabb8bff923ad06d047e6247d)