[2026-04-15 22:40:01,378][ INFO][PID:2782507] Marking build as starting [2026-04-15 22:40:01,462][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:01,470][ INFO][PID:2782507] VM allocation process starts [2026-04-15 22:40:01,548][ INFO][PID:2782507] Trying to allocate VM: ResallocHost, ticket_id=3141, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:40:06,560][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:11,564][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:16,565][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:21,569][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:26,572][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:31,578][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:36,583][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:41,586][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:46,588][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:51,594][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:40:56,599][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:01,600][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:06,602][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:11,605][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:16,609][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:21,610][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:26,613][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:31,617][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:36,620][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:41,624][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:46,629][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:51,631][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:41:56,635][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:01,637][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:06,642][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:11,648][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:16,652][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:21,657][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:26,661][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:31,664][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:36,666][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:41,669][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:46,670][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:51,672][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:42:56,675][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:01,679][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:06,681][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:11,684][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:16,689][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:19,854][ INFO][PID:2782507] Allocated host ResallocHost, ticket_id=3141, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147481_20260415_224010, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:43:19,855][ INFO][PID:2782507] Allocating ssh connection to builder [2026-04-15 22:43:19,857][ INFO][PID:2782507] Checking that builder machine is OK [2026-04-15 22:43:20,701][ INFO][PID:2782507] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:43:20,702][ ERROR][PID:2782507] Re-try request for task on 'ResallocHost, ticket_id=3141, hostname=2620:52:6:1161:dead:beef:cafe:c900, name=vmhost_p09_01_dev_02147481_20260415_224010, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:43:20,704][ INFO][PID:2782507] VM Release request [2026-04-15 22:43:20,720][ INFO][PID:2782507] Retry #1 (on other host) [2026-04-15 22:43:20,723][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:20,724][ INFO][PID:2782507] VM allocation process starts [2026-04-15 22:43:20,756][ INFO][PID:2782507] Trying to allocate VM: ResallocHost, ticket_id=3203, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:43:25,759][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:30,760][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:35,766][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:40,770][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:45,774][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:50,779][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:43:55,782][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:00,784][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:05,790][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:10,801][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:15,812][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:20,815][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:25,816][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:30,820][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:35,822][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:40,825][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:45,827][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:50,832][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:44:55,836][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:00,840][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:05,845][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:10,852][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:15,858][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:20,859][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:25,863][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:30,867][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:35,873][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:39,073][ INFO][PID:2782507] Allocated host ResallocHost, ticket_id=3203, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147490_20260415_224218, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:45:39,073][ INFO][PID:2782507] Allocating ssh connection to builder [2026-04-15 22:45:39,074][ INFO][PID:2782507] Checking that builder machine is OK [2026-04-15 22:45:40,167][ INFO][PID:2782507] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:45:40,168][ ERROR][PID:2782507] Re-try request for task on 'ResallocHost, ticket_id=3203, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147490_20260415_224218, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:45:40,169][ INFO][PID:2782507] VM Release request [2026-04-15 22:45:40,201][ INFO][PID:2782507] Retry #2 (on other host) [2026-04-15 22:45:40,204][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:40,211][ INFO][PID:2782507] VM allocation process starts [2026-04-15 22:45:40,248][ INFO][PID:2782507] Trying to allocate VM: ResallocHost, ticket_id=3253, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:45:43,337][ INFO][PID:2782507] Allocated host ResallocHost, ticket_id=3253, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147490_20260415_224218, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:45:43,338][ INFO][PID:2782507] Allocating ssh connection to builder [2026-04-15 22:45:43,344][ INFO][PID:2782507] Checking that builder machine is OK [2026-04-15 22:45:43,492][ INFO][PID:2782507] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:45:43,493][ ERROR][PID:2782507] Re-try request for task on 'ResallocHost, ticket_id=3253, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147490_20260415_224218, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:45:43,494][ INFO][PID:2782507] VM Release request [2026-04-15 22:45:43,579][ INFO][PID:2782507] Retry #3 (on other host) [2026-04-15 22:45:43,580][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:43,584][ INFO][PID:2782507] VM allocation process starts [2026-04-15 22:45:43,684][ INFO][PID:2782507] Trying to allocate VM: ResallocHost, ticket_id=3255, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:45:48,689][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:53,696][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:45:58,700][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:03,704][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:08,715][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:13,718][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:18,723][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:23,739][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:28,750][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:33,769][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:38,780][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:43,787][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:48,793][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:53,797][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:46:58,802][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:03,806][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:08,817][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:13,822][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:18,833][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:23,843][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:28,850][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:33,852][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:38,858][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:43,863][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:48,870][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:53,872][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:47:58,873][ INFO][PID:2782507] Checking for cancel request [2026-04-15 22:48:02,212][ INFO][PID:2782507] Allocated host ResallocHost, ticket_id=3255, hostname=44.222.63.84, name=aws_x86_64_spot_dev_02147506_20260415_224535, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 22:48:02,214][ INFO][PID:2782507] Allocating ssh connection to builder [2026-04-15 22:48:02,216][ INFO][PID:2782507] Checking that builder machine is OK [2026-04-15 22:48:02,817][ INFO][PID:2782507] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:48:02,825][ ERROR][PID:2782507] Re-try request for task on 'ResallocHost, ticket_id=3255, hostname=44.222.63.84, name=aws_x86_64_spot_dev_02147506_20260415_224535, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:48:02,833][ INFO][PID:2782507] VM Release request [2026-04-15 22:48:02,938][ ERROR][PID:2782507] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c900', '2620:52:6:1161:dead:beef:cafe:c901', '44.222.63.84'} [2026-04-15 22:48:02,952][WARNING][PID:2782507] Switching not-finished job state to 'failed' [2026-04-15 22:48:02,969][ INFO][PID:2782507] Worker failed build, took 481.58251547813416 [2026-04-15 22:48:02,978][ INFO][PID:2782507] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.stg.fedoraproject.org", "memory_reqs": null, "enable_net": true, "project_owner": "@copr", "project_name": "TEST1776291979813580745TestDeleteGroupBuild", "project_dirname": "TEST1776291979813580745TestDeleteGroupBuild", "submitter": "frostyx", "ended_on": 1776293282.952946, "started_on": 1776292801.3704305, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": null, "buildroot_pkgs": null, "task_id": "2925222", "build_id": 2925222, "package_name": "example", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 8, "source_json": "{\"type\": \"git\", \"clone_url\": \"https://github.com/fedora-copr/copr-test-hello.git\", \"subdirectory\": \"\", \"committish\": \"rpkg-util\", \"spec\": \"\", \"srpm_build_method\": \"rpkg\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@copr/TEST1776291979813580745TestDeleteGroupBuild--frostyx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [ "fedora-43" ], "distributions_in_project": [ "fedora-43" ], "destdir": "/var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild", "results_repo_url": "https://download.copr-dev.fedorainfracloud.org/results/@copr/TEST1776291979813580745TestDeleteGroupBuild", "result_dir": "02925222", "built_packages": "", "tags": [ "arch_noarch" ], "id": 2925222, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "TEST1776291979813580745TestDeleteGroupBuild", "vendor": "Fedora Project COPR (@copr/TEST1776291979813580745TestDeleteGroupBuild)" } } ] } [2026-04-15 22:48:03,099][ INFO][PID:2782507] Sending fedora-messaging bus message in build.end [2026-04-15 22:48:08,659][ INFO][PID:2782507] Compressing /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/builder-live.log by gzip [2026-04-15 22:48:08,680][ INFO][PID:2782507] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/builder-live.log' as PID 2801544 [2026-04-15 22:48:08,699][ INFO][PID:2782507] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/builder-live.log: No such file or directory [2026-04-15 22:48:08,705][ ERROR][PID:2782507] Unable to compress file /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/builder-live.log [2026-04-15 22:48:08,712][ INFO][PID:2782507] Compressing /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/backend.log by gzip [2026-04-15 22:48:08,726][ INFO][PID:2782507] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776291979813580745TestDeleteGroupBuild/srpm-builds/02925222/backend.log' as PID 2801545