Azure Cloud Service Web Role web pages do not load

问题

This is most likely going to be a very long post to try to fully explain how my Azure Cloud Service is deployed and what I have already tried to tackle this issue, so apologies and thanks in advance.

Framwork: 4.5.1
Azure SDK: 2.5
Visual Studio: 2013 Update 4

The Problem:

Upon publishing my Azure Cloud Service when I try to load a web page the browser will load continuously, never displaying the page or any error (no web error code or .Net error). This is the same behaviour whether I try to load the website after remoting onto the VM in Azure itself, from the external IP or from -cloudservicename-.cloudapp.net. This is the case for both release and debug configurations when deployed to Azure.

However the application works and runs absolutely fine in both Debug and Release modes when running locally on the Azure Emulator. There are no differences between my cloud and local .cscfg files, barring a NetworkConfiguration section for VPN config.

The Cloud Service deploys from Visual Studio without error and I can RDP onto the VMs and open IIS etc no problem, with the configuration of the website and certificates looking fine.

My Application

My Azure Cloud Service is made up of two roles, a Web Role that hosts the website and a Worker role that runs various background jobs.

This project has been going for a while and I have published several iterations to Azure in the past from different machines. This is my first time publishing from a new Windows 8.1 laptop, although this shouldn't make a difference. The configuration has changed very little since the previous release (one connection string added to the role). The only major thing that has changed is that I upgraded to VS2013 update 4 (from update 3) and Azure SDK 2.5 (from 2.4).

I can post the configuration files if required, however everything seems to be OK with them given the deployment does work.

Investigation:

I have taken numerous steps to try to investigate and solve the problem.

RDPing onto the server reveals no immidiate issues, there are no IIS entries in the event log and my own logging doesn't seem to have anything useful in it (more on this later).

I have also enabled Azure Diagnostics but looking at the returned logs nothing jumps out at me as being a log of the issue at hand.

The next thing I tried is attaching the remote debugger to the Azure VM, this works and I can debug the project. I placed breakpoints on all of the methods I handle in Global.asax.vb, including:

Application_BeginRequest
Application_AuthenticateRequest
Application_Error

Upon requesting a page Application_BeginRequest is hit immidiately, followed by Application_AuthenticateRequest, as expected. however following the code through Application_AuthenticateRequest leads to some strange behaviour.

In this code I validate a FormsAuthenticationCookie, which if found to be something would set HttpContext.Current.User and Thread.CurrentPrincipal. IN this case the cookie is nothing as the user has not logged in, meaning the Application_AuthenticateRequest method finishes. At this point I can continue to step through the .Net framework code and can see the successful completion of HttpApplication.ExecuteStep.

At this point I would expect either the master_Page_PreLoad method or one of the other pre-load events in the MasterPage or my custom implementation of Page to execute, however none of them are hit.

After leaving the process in debug for a number of minutes eventually Application_Error is hit, meaning my logging writes the exception that apparently caused the error. This rarely happens but when it does I have seen two different errors come out:

First Error in my logging:

ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.). Additional Information : The client was trying to communicate with the server: net.tcp://rolename.here:24233.. GUID: 9f85d178-968f-4908-bec4-d1a5f57bb819

at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ErrStatus errStatus, Guid trackingId, Exception responseException, Byte[][] payload, EndpointID destination) at Microsoft.ApplicationServer.Caching.DataCacheFactory.EstablishConnection(IEnumerable'1 servers, RequestBody request, Func'3 sendMessageDelegate, DataCacheReadyRetryPolicy retryPolicy) at Microsoft.ApplicationServer.Caching.SocketClientProtocol.Initialize(IEnumerable'1 servers) at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName, CreateNewCacheDelegate cacheCreationDelegate, DataCacheInitializationViaCopyDelegate initializeDelegate) at Microsoft.Web.DistributedCache.CacheHelpers.RunCacheCreationHooks(CacheConnectingEventArgs fetchingEventArgs, IDataCacheFactory dataCacheFactory, Object sender, EventHandler'1 fetchingHandler, EventHandler'1 fetchedHandler) at Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.CreateInternalProvider(IHttpRuntime httpRuntime, OutputCacheInitializationData initData, IDataCacheFactory dataCacheFactory, EventHandler'1 cacheFetching, EventHandler'1 cacheFetched) at Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.GetInternalProvider() at Microsoft.Web.DistributedCache.DistributedCacheOutputCacheProvider.Get(String key) at System.Web.Caching.OutputCache.Get(String key) at System.Web.Caching.OutputCacheModule.OnEnter(Object source, EventArgs eventArgs) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Having investigated this error on line I found a Microsoft source (sorry I no longer have the link) saying basically "It is supposed to happen sometimes, don't worry about it" however this doesn't feel right. I have not seen this error before in my development work but one of the other developers working ont he project has experienced it in debug recently. Has anybody experienced this or knows for certain we dont have to worry about it?

In an attempt to rule this out I deployed the service with caching turned off on the web role but this made no difference.

Here is my cache configuration in web.config.

  <!--To use the in-role flavor of Windows Azure Cache, set identifier to be the cache cluster role name -->
  <!--To use the Windows Azure Cache Service, set identifier to be the endpoint of the cache cluster -->
  <autoDiscover isEnabled="true" identifier="rolename.here" />

  <!--<localCache isEnabled="true" sync="TimeoutBased" objectCount="100000" ttlValue="300" />-->
  <!--Use this section to specify security settings for connecting to your cache. This section is not required if your cache is hosted on a role that is a part of your cloud service. -->
  <!--<securityProperties mode="Message" sslEnabled="true">
    <messageSecurity authorizationInfo="[Authentication Key]" />
  </securityProperties>-->
</dataCacheClient>

The second error that gets recorded in my logging is:

System.Web.HttpException (0x80004005): Server cannot append header after HTTP headers have been sent. at System.Web.HttpResponse.AppendHeader(String name, String value) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

This feels like a side effect rather than the cause of the issue but any comments are welcome.

Other things I have tried:

Created an configured a new Cloud Service in Azure - same result.
Added a new Azure Cloud Service Project to the solution and configured it from scratch - same result.

At this point I am totally lost as to what to do, I cant understand why it works perfectly in debug but not when deployed, and why there is no log shedding more light on the issue - at the moment I am not even sure what to search for to find people who have had a similar problem.

The final question all this boils down to is why are my pages not loading?

I'm going to boot my old Windows 7 box back up to deploy from there and see if there is any difference.

Please don't hesitate to ask if you need any more information.

Thanks for reading this and for any help you can give.