TFS2013: Build Warehouse Sync Job Failed

Question

I'm getting the follow failure error on 1 collection out of 6. The other 5 are doing this job without failing. I've tried to run it manually using WarehouseControlWebService, but it doesn't work. Any help on figure out how to fix this. I know the SQL side is working since the others are running it without an error.

Job JobProcessingStatus="Idle" Name="Build Warehouse Sync"

LastRun Result="Failed" EndTimeUtc="2016-05-26T01:13:03.253Z" ExecutionStartTimeUtc="2016-05-26T00:52:57.857Z" QueueTimeUtc="2016-05-26T00:52:57.603Z"

[Build Warehouse Sync]: ---> Microsoft.TeamFoundation.Warehouse.WarehouseException: TF221122: An error occurred running job Build Warehouse Sync for team project collection or Team Foundation server Collection2. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString() at Microsoft.TeamFoundation.Warehouse.WarehouseBatchedDataAccessComponent.FormatFactEntriesXml(Fact fact, IEnumerable 1 entries) at Microsoft.TeamFoundation.Warehouse.WarehouseBatchedDataAccessComponent.SaveFactEntries(Fact fact, IEnumerable 1 entries, Boolean allowUpdate) at Microsoft.TeamFoundation.Build.Adapter.WarehouseEntry.SaveFactEntries(WarehouseBatchedDataAccessComponent dac, WarehouseConfig config, IEnumerable 1 entries, Boolean allowUpdate) at Microsoft.TeamFoundation.Build.Adapter.BuildBatch.SaveBatch() at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.ProcessAndUploadBuildData() at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.MakeDataChanges() at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension 1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension` 1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) --- End of inner exception stack trace ---

Answer 1

You may try to rebuild the TFS data warehouse and cube to see whether the issue persists. Please notice depending on the amount of data in the data warehouse, the rebuild operation can take several hours to finish, during which time reports are not available.

Answer 2

With Microsoft's help we had to modify the batch size of the job. Bellow are my notes from what we did. After adjusting the size a few times and allowing it to run it cleared.

Use this query in the configuration database to determine the HostID of the collection Database that you want to modify

Select *

from tbl_ServiceHost

sample:

hostid name 185C91E1-2EB5-4C9C-AE97-26B757DE951B testnongenerated AD455D2B-7A84-4BF6-A8F2-594895066472 DefaultCollection 8F2ECA36-47BE-4244-B4C7-E2AABC7693FE TEAM FOUNDATION 8D9C0BCE-19AB-425B-8D27-ECDE6573A493 New

So to use DefaultCollection we need the HostId: AD455D2B-7A84-4BF6-A8F2-594895066472

Enter the HostID into this Insert Query to run in the Warehouse Database:

INSERT INTO [_PropertyBag] ( [Property_Scope], [Property_Key], [Property_Value] ) VALUES ( ' AD455D2B-7A84-4BF6-A8F2-594895066472', -- HostID from tbl_ServiceHost '/Adapter/Config/TeamBuild/BatchSize', -- Property to Update, this is the key for the Build Warehouse Sync '100' -- New Batch Size (default 2000)
)

We want to take this down to a very small number then allow it to run and make sure it is running successfully, if not we can lower it to 1 and try that, if so we can (at your discretion) move it back up if you want to increase processing speed.

Best to stop all Warehouse Sync Jobs and run the 1 Job having the issue. Will take time for it to clear issue. Make sure DB back created of TFS_Warehouse.

This was used to fix the following error:

Build Warehouse Sync: ---> Microsoft.TeamFoundation.Warehouse.WarehouseException: TF221122: An error occurred running job Build Warehouse Sync for team project collection or Team Foundation server AS. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString() at Microsoft.TeamFoundation.Warehouse.WarehouseBatchedDataAccessComponent.FormatFactEntriesXml(Fact fact, IEnumerable 1 entries) at Microsoft.TeamFoundation.Warehouse.WarehouseBatchedDataAccessComponent.SaveFactEntries(Fact fact, IEnumerable 1 entries, Boolean allowUpdate) at Microsoft.TeamFoundation.Build.Adapter.WarehouseEntry.SaveFactEntries(WarehouseBatchedDataAccessComponent dac, WarehouseConfig config, IEnumerable 1 entries, Boolean allowUpdate) at Microsoft.TeamFoundation.Build.Adapter.BuildBatch.SaveBatch() at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.ProcessAndUploadBuildData() at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.MakeDataChanges() at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension 1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension` 1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) — End of inner exception stack trace ---