.net 4.61 httpwebrequest on all .tv domains fail, especially https://www.tvone.tv

前端未结

关注

 1  470

I am using .net 4.61 and doing a httpwebrequest with VB on https://www.tvone.tv and it fails to connect to server every time. Any other lookup on any other domain other than .tv

相关标签:

1条回答

独厮守ぢ

2021-01-25 23:21

One of the sites you mentioned had a Https: schema in the URI.
This implies that a security protocol is used. Upon inspection, this site uses TLS 1.2 security protocol. So you have to enable it using ServicePointManager, setting: ServicePointManager.SecurityProtocol= SecurityProtocolType.Tls12.

Also, most of the time, sites use Cookies (or have to use them, because external resources require them to. Google tools, for example).
You have to provide means to accept, store and exchange them.
This is accomplished instantiating a CookieContainer Object and passing it to the HttpWebRequest using its .CookieContainer() Property

So I tested all of them to see what was happening behind the curtains (see the results).

One of them is more interesting than the others (https://www.tvone.tv). It's one of those that don't accept other schema than Https:. If you change it, it will redirect your request back to it (forcing security with Tls 1.2 certificate exchange). But it doesn't (the home page, at least) use cookies.
Since I think it's somewhat interesting, I'm posting the code used for testing. If, in the Headers setup, you set .AllowAutoRedirect = False, you can follow the redirection and see what it does.

Host URI: https://tvone.tv/
IP Address: 45.79.161.20
Response Status: OK Http Protocol: Http/1.1
Cookies n°: 0 Security Protocol: Tls 1.2

Try this with Https: schema
Host URI: http://www.ustream.tv/
IP Address: 199.66.238.212
Response Status: OK Http Protocol: Http/1.1
Cookies n°: 2 Security Protocol: N/A

Host URI: http://www.fyi.tv/
IP Address: 151.101.14.168
Response Status: OK Http Protocol: Http/1.1
Cookies n°: 1 Security Protocol: N/A

This is the code used for testing.
(It's a redacted/translated redux of a library of mine, so you don't have to worry about stepping on anyone's toes if you use it).

The main function, HTTP_GetWebPageAsync(), can be called like this:
(The object passed is returned with its properties filled with the data shown in the results above, plus the whole Html page (in Public Property PayLoad) of the response).

Public HttpReqObject As HttpReqObj = New HttpReqObj

Dim MaxWaitTimeOut As Integer = 30  'Seconds
HttpReqObject.SiteURL = "https://tvone.tv/"
HttpReqObject = Await HTTP_GetWebPageAsync(HttpReqObject, MaxWaitTimeOut)

Imports System.IO
Imports System.Net
Imports System.Net.Security
Imports System.Reflection
Imports System.Security
Imports System.Security.Authentication
Imports System.Security.Cryptography.X509Certificates
Imports System.Text

Private Const COR_E_INVALIDOPERATION As Int32 = &H80131509

Public Class HttpReqObj
    Public Property SiteURL As String
    Public Property StatusCode As HttpStatusCode
    Public Property PayLoad As String
    Public Property Cookies As CookieContainer
    Public Property HostAddress As IPAddress()
    Public Property HttpProtocol As String
    Public Property SslProtocol As SslProtocols
End Class

Public Async Function HTTP_GetWebPageAsync(HttpSite As HttpReqObj, ReqTimeOut As Integer) As Task(Of HttpReqObj)
    Dim _HttpRequest As HttpWebRequest
    Dim _Cookiejar As New CookieContainer()
    Dim _StatusCode As HttpStatusCode
    Dim _RedirectURL As String = String.Empty
    Dim _Referer As String = String.Empty
    Dim _Payload As String = String.Empty
    Dim _MaxAwaitableTimeout As Integer = ReqTimeOut
    Dim _MaxHops As Integer = 40

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 Or
                                           SecurityProtocolType.Tls11 Or
                                           SecurityProtocolType.Tls12
    ServicePointManager.ServerCertificateValidationCallback = AddressOf TlsValidationCallback

    Try
        _HttpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
        HttpSite.HostAddress = Await Dns.GetHostAddressesAsync(_HttpRequest.Host)
        HTTP_RequestHeadersInit(_HttpRequest, _Cookiejar, "")
        _HttpRequest.Method = WebRequestMethods.Http.Get
        Using httpResponse As HttpWebResponse = CType(Await _HttpRequest.GetResponseAsync(), 
                                                            HttpWebResponse)
            _StatusCode = httpResponse.StatusCode
            _Payload = If(_StatusCode = HttpStatusCode.OK, ProcessResponse(httpResponse, HttpSite), String.Empty)
            _RedirectURL = URIFromResponseLocation(httpResponse).ToString()
            HttpSite.HttpProtocol = "Http/" + _HttpRequest.ProtocolVersion.ToString()
        End Using


        'On Page Redirection, follow redirections [_MaxHops] hops and [_MaxAwaitableTimeout] time
        If (_RedirectURL <> HttpSite.SiteURL) Or (_StatusCode = HttpStatusCode.Moved) OrElse
                                                 (_StatusCode = HttpStatusCode.Found) OrElse
                                                 (_StatusCode = HttpStatusCode.RedirectMethod) Then

               If (_RedirectURL <> HttpSite.SiteURL) Then
                   Dim _SW As Stopwatch = New Stopwatch()
                   _SW.Start()

                   _Referer = HttpSite.SiteURL
                   HttpSite.SiteURL = _RedirectURL
                   _HttpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
                   _HttpRequest.Method = WebRequestMethods.Http.Get
                   HTTP_RequestHeadersInit(_HttpRequest, _Cookiejar, _Referer)

                   Dim _hops As Integer = 1
                   Do
                        Using httpResponse As HttpWebResponse = CType(Await _HttpRequest.GetResponseAsync(), 
                                                                            HttpWebResponse)
                            _Payload = ProcessResponse(httpResponse, HttpSite)
                            _StatusCode = httpResponse.StatusCode
                            _RedirectURL = URIFromResponseLocation(httpResponse).ToString()
                        End Using

                        If _RedirectURL <> HttpSite.SiteURL Then
                            If (_StatusCode = HttpStatusCode.Moved) OrElse
                               (_StatusCode = HttpStatusCode.Found) OrElse
                               (_StatusCode = HttpStatusCode.RedirectMethod) Then

                                HttpSite.SiteURL = _RedirectURL
                                _HttpRequest = WebRequest.CreateHttp(HttpSite.SiteURL)
                                HTTP_RequestHeadersInit(_HttpRequest, _Cookiejar, _Referer)
                                _hops += 1
                            End If
                        End If

                        If _SW.Elapsed.Seconds > _MaxAwaitableTimeout Then
                            _StatusCode = HttpStatusCode.RequestTimeout
                            Exit Do
                        End If

                   Loop While (_StatusCode <> HttpStatusCode.OK) AndAlso (_hops < _MaxHops)
                _SW.Stop()
                End If
       End If


    Catch exW As WebException
        _StatusCode = If(exW.Response IsNot Nothing, CType(exW.Response, HttpWebResponse).StatusCode,
                                                     CType(exW.Status, HttpStatusCode))
        _Payload = String.Empty
    Catch exS As System.Exception
        If exS.HResult = COR_E_INVALIDOPERATION Then
            '_StatusCode = If(InternetConnection() > 0, CType(WebExceptionStatus.NameResolutionFailure, HttpStatusCode),
            '                                           CType(WebExceptionStatus.ConnectFailure, HttpStatusCode))
            _StatusCode = CType(WebExceptionStatus.ConnectFailure, HttpStatusCode)
        Else
            _StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)
        End If

        _Payload = String.Empty
    Finally
        ServicePointManager.ServerCertificateValidationCallback = Nothing
    End Try

    HttpSite.Cookies = _Cookiejar
    HttpSite.PayLoad = _Payload
    HttpSite.StatusCode = _StatusCode

    Return HttpSite

End Function


Private Function TlsValidationCallback(sender As Object, CACert As X509Certificate, CAChain As X509Chain, ssl_PolicyErrors As SslPolicyErrors) As Boolean

    If ssl_PolicyErrors = SslPolicyErrors.None Then
        Return True
    End If

    Dim _Certificate As New X509Certificate2(CACert)
    'Dim _CACert As New X509Certificate2("./ca.cert")  'Add your certificate here
    'CAChain.ChainPolicy.ExtraStore.Add(_CACert)
    CAChain.Build(_Certificate)
    For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
        If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
           (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
            Return False
        End If
    Next
    Return True
End Function

Private Function ExtractSslProtocol(stream As Stream) As SslProtocols
    Dim bindingFlags__1 As BindingFlags = BindingFlags.Instance Or BindingFlags.Public Or BindingFlags.NonPublic Or BindingFlags.Static

    Dim _objConnection = stream.[GetType]().GetField("m_Connection", bindingFlags__1).GetValue(stream)
    Dim _objTlsStream = _objConnection.[GetType]().GetProperty("NetworkStream", bindingFlags__1).GetValue(_objConnection)
    Dim _objSslState = _objTlsStream.[GetType]().GetField("m_Worker", bindingFlags__1).GetValue(_objTlsStream)
    Return CType(_objSslState.[GetType]().GetProperty("SslProtocol", bindingFlags__1).GetValue(_objSslState), SslProtocols)
End Function


Private Sub HTTP_RequestHeadersInit(ByRef HttpReq As HttpWebRequest, CookieJar As CookieContainer, Referer As String)

    HttpReq.ServicePoint.MaxIdleTime = 30000
    HttpReq.ServicePoint.Expect100Continue = False
    HttpReq.Timeout = 30000
    HttpReq.CookieContainer = CookieJar
    HttpReq.KeepAlive = True
    HttpReq.AllowAutoRedirect = False
    HttpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
    HttpReq.Referer = Referer
    HttpReq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
    HttpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    HttpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.8,en;q=0.5")
    HttpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
    HttpReq.Headers.Add(HttpRequestHeader.CacheControl, "max-age=0")

End Sub

Private Function ProcessResponse(Response As HttpWebResponse, HttpSite As HttpReqObj) As String

    Dim _html As String
    Dim _encoding As Encoding

    Try
        _encoding = If(Response.CharacterSet.Length > 0,
                       Encoding.GetEncoding(Response.CharacterSet),
                       Encoding.GetEncoding("utf-8"))

        Using _stream As Stream = Response.GetResponseStream()
        HttpSite.SslProtocol = ExtractSslProtocol(_stream)
            Using _memStream As New MemoryStream()
                If Response.ContentEncoding.ToLower().Contains("gzip") Then
                    Using _gzipStream As New Compression.GZipStream(_stream,
                                             Compression.CompressionMode.Decompress)
                         _gzipStream.CopyTo(_memStream)
                    End Using
                ElseIf Response.ContentEncoding.ToLower().Contains("deflate") Then
                    Using _deflStream As New Compression.DeflateStream(_stream,
                                             Compression.CompressionMode.Decompress)
                         _deflStream.CopyTo(_memStream)
                    End Using
                Else
                    _stream.CopyTo(_memStream)
                End If

                _memStream.Position = 0
                Using _reader As New StreamReader(_memStream, _encoding)
                    _html = _reader.ReadToEnd().Trim()
                End Using
            End Using

        End Using
    Catch _Ex As Exception
        Return String.Empty
    End Try

    Return _html

End Function

Private Function URIFromResponseLocation(RefResponse As HttpWebResponse) As System.Uri
    Dim _uri As System.Uri
    Dim _location As String = RefResponse.Headers("Location")

    Try
        If Uri.IsWellFormedUriString(_location, UriKind.Absolute) Then
            _uri = New Uri(_location, UriKind.Absolute)
        Else
            Dim _host_uri As String = RefResponse.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
                                                                            UriFormat.Unescaped) + _location
            If Uri.IsWellFormedUriString(_host_uri, UriKind.Absolute) Then
                _uri = New Uri(_host_uri)
            Else
                _uri = New Uri(RefResponse.ResponseUri.GetComponents(UriComponents.Scheme,
                                                                     UriFormat.Unescaped) +
                                                                     RefResponse.ResponseUri.Host + _location)
            End If
        End If
    Catch _Ex As Exception
        _uri = New Uri(_location, UriKind.Relative)
    End Try

    Return _uri

End Function

0 讨论(0)