Issues when deploying solution to Service Fabric cluster

Question

All,

I seem to be unable to deploy an application that runs perfectly on my local cluster to a cluster in Azure.

Strange thing is that I was able to deploy this application in the past, but seem unable to now. Upon deployment of the application it seems that my stateless services are first deployed without any problems, but my statefull services and actors seem to be unable to locate suitable nodes for replicas, as I seem to be receiving the following warning (through the service fabric explorer, when selecting the application):

Unhealthy event: SourceId='System.PLB', Property='ServiceReplicaUnplacedHealth_Primary_8f41b6fb-3c6f-4916-9f85-27da0171ae6d', HealthState='Warning', ConsiderWarningAsError=false.
The Load Balancer was unable to find a placement for one or more of the Service's Replicas:
fabric:/Namespace/ActorService Primary Partition 8f41b6fb-3c6f-4916-9f85-27da0171ae6d could not be placed, possibly, due to the following constraints and properties:  
Placement Constraint: N/A
Depended Service: N/A

Constraint Elimination Sequence:
NodeBlockList eliminated 5 possible node(s) for placement -- 0/5 node(s) remain.

Upon further investigation, I also encountered the following error:

Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint'.
ServiceManifestName='ActorPkg', NodeName='_nt1vm_4',AggregatedHealthState='Error'.

When checking all log files in blob storage, as well as the traces, I also found the following log statements:

2016-5-12 12:54:15.176,Warning,1516,2528,Hosting.ProcessActivationManager,Application service 66933b89-78e9-4590-8971-d0cad139d956 with processId 3240 for parent process 310a2a579b75d9847ba5a43d6b365d7c terminated with exit code 2148734720
2016-5-12 12:54:15.176,Warning,1516,2528,Hosting._Nodes_ProcessUnexpectedTermination@310a2a579b75d9847ba5a43d6b365d7c,ServiceHostProcess: Process.exe for ApplicationId 66933b89-78e9-4590-8971-d0cad139d956 terminated unexpectedly with exit code 2148734720 on node id 310a2a579b75d9847ba5a43d6b365d7c
2016-5-12 12:54:15.176,Informational,1516,2528,Hosting.ProcessActivationManager,Sending ApplicationService host down notification for apphost 66933b89-78e9-4590-8971-d0cad139d956, node 310a2a579b75d9847ba5a43d6b365d7c
2016-5-12 12:54:15.176,Warning,2888,2556,Hosting.ApplicationHostManager@310a2a579b75d9847ba5a43d6b365d7c:131075269508464788,Processing termination of activated Application Host: HostId=66933b89-78e9-4590-8971-d0cad139d956, ExitCode=2148734720
2016-5-12 12:54:15.176,Warning,2888,2556,IPC.ServerFailedToRemoveClient,527e532390 client 66933b89-78e9-4590-8971-d0cad139d956 not found
2016-5-12 12:54:15.176,Informational,2888,2556,Hosting.ApplicationHostManager@310a2a579b75d9847ba5a43d6b365d7c:131075269508464788,RemoveActivationTableEntry: ErrorCode=S_OK, HostId=66933b89-78e9-4590-8971-d0cad139d956
2016-5-12 12:54:15.176,Informational,1516,2528,Transport.Enqueue,adb77036b0 enqueue ef74d99d-1a29-4c34-b6d1-1f59542a42d8:1926 False  319B @ qsize 0/0B
2016-5-12 12:54:15.176,Informational,2864,2556,RA.HostingProcessClosedEvent_Hosting@310a2a579b75d9847ba5a43d6b365d7c:131075269508464788,RA on node 310a2a579b75d9847ba5a43d6b365d7c:131075269508464788 performing AppHost down processing for apphost: 66933b89-78e9-4590-8971-d0cad139d956/
2016-5-12 12:54:15.176,Informational,2864,2556,RA.MultipleFTWorkBegin_MultipleFT@310a2a579b75d9847ba5a43d6b365d7c:131075269508464788,Multiple FT Work Begin [Activity: AppHostClosed 66933b89-78e9-4590-8971-d0cad139d956]. FTs 36
2016-5-12 12:54:15.176,Informational,2888,2556,Hosting.HostingHealthManager@310a2a579b75d9847ba5a43d6b365d7c:131075269508464788,ServicePackage ReportHealth: HealthReport(fabric:/Namespace+ActorPkg+310a2a579b75d9847ba5a43d6b365d7c instance=131075312250519674 System.Hosting CodePackageActivation:Code:EntryPoint Error ttl=Infinite sn=131075312551846737 There was an error during CodePackage activation.The service host terminated with exit code:2148734720 removeWhenExpired=false 2016-05-12 12:54:15.184 priority NotAssigned)

If anybody has any ideas or suggestions as to why my application can no longer be deployed on the cloud that would be greatly appreciated!

Kind regards

Answer 1

In response to The Muffin Man's comment, I've removed my answer from the question and placed it here (and accepted it as a solution):

After figuring out how to turn out additional diagnostic information. (Via cloud explorer, navigate to the Virtual Machine Scale Sets and then select the scale set. In my case 'nt1vm' and choose Update Diagnostics... via Actions and turn on all logging)

I came upon this line (in the WADWindowsEventLogsTable):

.NET Runtime version : 4.0.30319.34209 - This application could not be started.This application requires one of the following versions of the .NET Framework: .NETFramework,Version=v4.6.1Do you want to install this .NET Framework version now?

So I'm downgrading my application to use .NET 4.5.2.

UPDATE SOLVED: After downgrading to .NET 4.5.2 my application deployed perfectly. Thanks for this post: Hosting console application in public service fabric cluster to let me try this.

Answer 2

I had this same 2148734720 error happen on my Service Fabric deployment too. My SF App was targeting .net 4.6.1, but the SF node did not have .net 4.6.X installed on it.

After installing .net 4.6.2 on each of the deployment SF nodes (and rebooting), this error was fixed.

Answer 3

If you're using .Net core with a non en-US version of Windows you may be experiencing this issue with framework versions earlier than 2.0.3: https://github.com/Azure/service-fabric-issues/issues/643

As the GitHub discussion says, the solution is to upgrade to a new version of .Net Core. However, even after doing that you may still experience the problem if the new version of the framework isn't being picked up.

You can workaround that by explicitly specifying the version in your .csproj:

 <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <RuntimeFrameworkVersion>2.0.5</RuntimeFrameworkVersion>
    <IsServiceFabricServiceProject>True</IsServiceFabricServiceProject>
 </PropertyGroup>