slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

jeffschwMSFT · 2024-01-24T15:45:11Z

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
945311	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
2637853	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere main	Log
944957	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#60263
944670	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#110622
2637610	dotnet-efcore	macOS	Log
2637415	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
943797	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
943759	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46527
943627	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
2636981	dotnet-arcade-validation	MacOS Build_Release	Log
943388	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#59952
943030	dotnet/performance	Performance imagesharp windows 22H2 x64 Open 9.0	Log
2636745	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere 8.0	Log
2636743	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere 9.0	Log
2636739	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere main	Log
942983	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#59952
942887	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46602
942871	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46562
942773	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46575
942737	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#46467
942780	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#43073
942777	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46601
942749	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46397
942747	dotnet/sdk	OSX_x64	Log	dotnet/sdk#46397
942670	maestro-auth-test/build-result-analysis-test	HelixTests	Log	maestro-auth-test/build-result-analysis-test#1903
942629	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46603
942628	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46604
942608	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111598
942617	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46602
942435	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46598
942399	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46583
942597	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46601
942583	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46600
942575	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46518
942572	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46558
942524	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46290
942510	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46361
942508	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46562
942227	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46593
942498	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46233
942495	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46591
942193	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46585
942472	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46596
942230	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46592
942215	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46589
942197	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46502
942449	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46529
942190	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46584
942443	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46599
942421	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46397
942412	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46576
942411	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46577
942410	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46578
942408	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46579
942407	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46597
942174	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46583
941910	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46203
941895	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46086
942393	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46285
942390	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46359
942080	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#46302
942081	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46301
942079	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46317
942054	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46543
942052	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46545
941918	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46571
942099	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46425
942257	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46527
942256	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46527
942240	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46595
942235	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46507
942233	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46594
942226	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46591
942218	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46546
942214	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46590
942211	dotnet/sdk	Darwin_AoT_Tests Build_Release		dotnet/sdk#46505
942203	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46588
942202	dotnet/sdk	Darwin Build_Release	Log	dotnet/sdk#46587
942199	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46586
942157	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log
942149	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log
942119	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log
942117	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46582
940696	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46296
942069	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46386
942065	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46415
942063	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46570
942062	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46416
942056	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#46529
942049	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46573
941781	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46575
941770	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46568
941980	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#45969
941958	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46581
941637	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#46572
941920	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46359
941916	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46285
941911	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46580
941907	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#46197
941898	dotnet/sdk	Darwin Build_Release	Log
Displaying 100 of 1431 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	261	1431

The text was updated successfully, but these errors were encountered:

lewing · 2024-02-09T01:11:11Z

🤔

nagilson · 2024-10-03T16:50:21Z

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

ivanpovazan · 2025-01-27T11:29:19Z

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

garath · 2025-01-27T20:24:33Z

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

garath · 2025-01-28T22:56:54Z

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

jeffschwMSFT added First Responder Known Build Error labels Jan 24, 2024

lewing mentioned this issue Feb 8, 2024

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" #1886

Open

3 tasks

AaronRobinsonMSFT mentioned this issue Dec 10, 2024

Fix test under JIT stress dotnet/runtime#110538

Merged

garath self-assigned this Jan 27, 2025

garath removed their assignment Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

jeffschwMSFT commented Jan 24, 2024 •

edited by build-analysis bot

Loading

lewing commented Feb 9, 2024

nagilson commented Oct 3, 2024

ivanpovazan commented Jan 27, 2025

garath commented Jan 27, 2025

garath commented Jan 28, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

Comments

jeffschwMSFT commented Jan 24, 2024 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Known issue core information

Release Note Category

Release Note Description

Additional information about the issue reported

Known issue validation

Report

Summary

lewing commented Feb 9, 2024

nagilson commented Oct 3, 2024

ivanpovazan commented Jan 27, 2025

garath commented Jan 27, 2025

garath commented Jan 28, 2025

jeffschwMSFT commented Jan 24, 2024 •

edited by build-analysis bot

Loading