[2026-04-15 23:14:52,138][ INFO][PID:2868793] Marking build as starting [2026-04-15 23:14:52,241][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:14:52,245][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:14:52,262][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4003, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:14:57,271][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:02,274][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:07,278][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:12,281][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:17,283][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:22,289][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:27,293][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:32,296][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:37,299][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:42,302][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:47,311][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:52,316][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:15:57,322][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:02,326][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:07,333][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:12,335][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:17,338][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:22,342][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:27,344][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:32,346][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:37,352][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:40,631][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4003, hostname=44.212.29.235, name=aws_x86_64_spot_dev_02147617_20260415_231439, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:16:40,643][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:16:40,653][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:16:41,097][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:16:41,098][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4003, hostname=44.212.29.235, name=aws_x86_64_spot_dev_02147617_20260415_231439, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:16:41,104][ INFO][PID:2868793] VM Release request [2026-04-15 23:16:41,131][ INFO][PID:2868793] Retry #1 (on other host) [2026-04-15 23:16:41,135][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:41,141][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:16:41,178][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4049, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:16:46,183][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:51,189][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:16:56,195][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:01,197][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:06,200][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:11,203][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:16,207][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:21,211][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:26,218][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:31,221][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:36,224][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:41,228][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:46,230][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:51,236][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:17:56,238][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:01,240][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:06,244][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:09,539][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4049, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:09,541][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:09,542][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:10,435][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:10,436][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4049, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:10,437][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:10,480][ INFO][PID:2868793] Retry #2 (on other host) [2026-04-15 23:18:10,483][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:10,488][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:10,532][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4084, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:13,591][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4084, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:13,592][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:13,595][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:13,721][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:13,723][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4084, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:13,725][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:13,747][ INFO][PID:2868793] Retry #3 (on other host) [2026-04-15 23:18:13,748][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:13,750][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:13,801][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4085, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:16,874][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4085, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:16,876][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:16,878][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:17,022][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:17,024][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4085, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:17,028][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:17,058][ INFO][PID:2868793] Retry #4 (on other host) [2026-04-15 23:18:17,060][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:17,065][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:17,103][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4087, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:22,114][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:23,177][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4087, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:23,179][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:23,180][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:23,321][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:23,322][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4087, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:23,325][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:23,350][ INFO][PID:2868793] Retry #5 (on other host) [2026-04-15 23:18:23,353][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:23,358][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:23,389][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4088, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:26,443][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4088, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:26,445][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:26,448][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:26,580][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:26,581][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4088, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:26,584][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:26,657][ INFO][PID:2868793] Retry #6 (on other host) [2026-04-15 23:18:26,660][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:26,665][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:26,746][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4089, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:29,832][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4089, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:29,833][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:29,837][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:29,977][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:29,978][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4089, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:29,983][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:30,004][ INFO][PID:2868793] Retry #7 (on other host) [2026-04-15 23:18:30,006][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:30,011][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:30,036][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4091, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:33,070][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4091, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:33,072][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:33,073][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:33,198][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:33,199][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4091, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:33,200][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:33,212][ INFO][PID:2868793] Retry #8 (on other host) [2026-04-15 23:18:33,212][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:33,215][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:33,252][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4092, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:36,302][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4092, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:36,303][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:18:36,304][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:18:36,427][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:18:36,428][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4092, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147618_20260415_231441, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:18:36,428][ INFO][PID:2868793] VM Release request [2026-04-15 23:18:36,438][ INFO][PID:2868793] Retry #9 (on other host) [2026-04-15 23:18:36,440][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:36,443][ INFO][PID:2868793] VM allocation process starts [2026-04-15 23:18:36,484][ INFO][PID:2868793] Trying to allocate VM: ResallocHost, ticket_id=4093, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:18:41,491][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:46,495][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:51,496][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:18:56,499][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:01,502][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:06,504][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:11,509][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:16,510][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:21,517][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:26,522][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:31,525][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:36,530][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:41,532][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:46,539][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:51,541][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:19:56,547][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:01,549][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:06,552][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:11,557][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:16,567][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:21,572][ INFO][PID:2868793] Checking for cancel request [2026-04-15 23:20:24,675][ INFO][PID:2868793] Allocated host ResallocHost, ticket_id=4093, hostname=3.235.56.29, name=aws_x86_64_spot_dev_02147634_20260415_231841, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:20:24,676][ INFO][PID:2868793] Allocating ssh connection to builder [2026-04-15 23:20:24,678][ INFO][PID:2868793] Checking that builder machine is OK [2026-04-15 23:20:25,038][ INFO][PID:2868793] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:20:25,043][ ERROR][PID:2868793] Re-try request for task on 'ResallocHost, ticket_id=4093, hostname=3.235.56.29, name=aws_x86_64_spot_dev_02147634_20260415_231841, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:20:25,048][ INFO][PID:2868793] VM Release request [2026-04-15 23:20:25,076][ ERROR][PID:2868793] Backend process error: Three host tried without success: {'3.235.56.29', '44.212.29.235', '2620:52:6:1161:dead:beef:cafe:c900'} [2026-04-15 23:20:25,082][WARNING][PID:2868793] Switching not-finished job state to 'failed' [2026-04-15 23:20:25,084][ INFO][PID:2868793] Worker failed build, took 332.9541621208191 [2026-04-15 23:20:25,087][ INFO][PID:2868793] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.stg.fedoraproject.org", "memory_reqs": null, "enable_net": true, "project_owner": "@copr", "project_name": "TEST1776292405247001007Timeout", "project_dirname": "TEST1776292405247001007Timeout", "submitter": "frostyx", "ended_on": 1776295225.0827909, "started_on": 1776294892.1286287, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": null, "buildroot_pkgs": null, "task_id": "2925263", "build_id": 2925263, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 1, "source_json": "{\"url\": \"https://github.com/fedora-copr/copr-test-sources/raw/main/hello-2.8-1.src.rpm\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@copr/TEST1776292405247001007Timeout--frostyx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [], "distributions_in_project": [ "fedora-43" ], "destdir": "/var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout", "results_repo_url": "https://download.copr-dev.fedorainfracloud.org/results/@copr/TEST1776292405247001007Timeout", "result_dir": "02925263", "built_packages": "", "tags": [ "arch_noarch" ], "id": 2925263, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "TEST1776292405247001007Timeout", "vendor": "Fedora Project COPR (@copr/TEST1776292405247001007Timeout)" } } ] } [2026-04-15 23:20:25,171][ INFO][PID:2868793] Sending fedora-messaging bus message in build.end [2026-04-15 23:20:26,159][ INFO][PID:2868793] Compressing /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/builder-live.log by gzip [2026-04-15 23:20:26,163][ INFO][PID:2868793] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/builder-live.log' as PID 2882204 [2026-04-15 23:20:26,165][ INFO][PID:2868793] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/builder-live.log: No such file or directory [2026-04-15 23:20:26,167][ ERROR][PID:2868793] Unable to compress file /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/builder-live.log [2026-04-15 23:20:26,169][ INFO][PID:2868793] Compressing /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/backend.log by gzip [2026-04-15 23:20:26,172][ INFO][PID:2868793] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776292405247001007Timeout/srpm-builds/02925263/backend.log' as PID 2882205