I think we need to be REALLY careful about making assertions about an origin in a response, given things like pretty-bad-proxy.

Section 3 says:

Alt-Svc MAY occur in any HTTP response message, regardless of the status code.

That's way too broad. For example, what if it's in response to a CONNECT -- will clients associate it with the origin?

I think the resolution of this will need to reference this:
http://httpwg.github.io/specs/rfc7231.html#identification