[2026-04-15 23:25:15,574][ INFO][PID:2893799] Marking build as starting [2026-04-15 23:25:15,679][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:15,684][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:25:15,707][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4185, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:25:20,715][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:25,718][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:30,719][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:35,722][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:40,730][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:45,740][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:50,745][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:25:55,748][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:00,751][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:05,754][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:10,758][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:15,765][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:20,775][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:25,777][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:30,791][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:35,796][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:40,810][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:44,029][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4185, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:44,030][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:26:44,032][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:26:44,631][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:26:44,632][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4185, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:26:44,632][ INFO][PID:2893799] VM Release request [2026-04-15 23:26:44,648][ INFO][PID:2893799] Retry #1 (on other host) [2026-04-15 23:26:44,649][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:44,650][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:26:44,694][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4210, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:47,741][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4210, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:47,742][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:26:47,742][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:26:47,812][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:26:47,813][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4210, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:26:47,813][ INFO][PID:2893799] VM Release request [2026-04-15 23:26:47,824][ INFO][PID:2893799] Retry #2 (on other host) [2026-04-15 23:26:47,824][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:47,825][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:26:47,868][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4211, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:50,908][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4211, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:50,909][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:26:50,909][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:26:50,979][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:26:50,980][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4211, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:26:50,981][ INFO][PID:2893799] VM Release request [2026-04-15 23:26:50,993][ INFO][PID:2893799] Retry #3 (on other host) [2026-04-15 23:26:50,994][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:50,996][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:26:51,041][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4212, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:54,080][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4212, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:54,081][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:26:54,082][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:26:54,152][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:26:54,153][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4212, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:26:54,154][ INFO][PID:2893799] VM Release request [2026-04-15 23:26:54,165][ INFO][PID:2893799] Retry #4 (on other host) [2026-04-15 23:26:54,167][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:54,169][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:26:54,203][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4213, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:57,245][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4213, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:26:57,246][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:26:57,248][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:26:57,345][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:26:57,346][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4213, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:26:57,348][ INFO][PID:2893799] VM Release request [2026-04-15 23:26:57,396][ INFO][PID:2893799] Retry #5 (on other host) [2026-04-15 23:26:57,397][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:26:57,399][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:26:57,434][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4214, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:00,479][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4214, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:00,480][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:27:00,481][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:27:00,555][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:27:00,556][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4214, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:27:00,557][ INFO][PID:2893799] VM Release request [2026-04-15 23:27:00,575][ INFO][PID:2893799] Retry #6 (on other host) [2026-04-15 23:27:00,577][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:00,580][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:27:00,642][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4215, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:03,704][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4215, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:03,705][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:27:03,705][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:27:03,796][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:27:03,797][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4215, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:27:03,798][ INFO][PID:2893799] VM Release request [2026-04-15 23:27:03,824][ INFO][PID:2893799] Retry #7 (on other host) [2026-04-15 23:27:03,825][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:03,828][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:27:03,873][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4216, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:06,948][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4216, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:06,949][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:27:06,949][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:27:07,049][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:27:07,051][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4216, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147656_20260415_232353, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:27:07,052][ INFO][PID:2893799] VM Release request [2026-04-15 23:27:07,086][ INFO][PID:2893799] Retry #8 (on other host) [2026-04-15 23:27:07,092][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:07,093][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:27:07,136][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4217, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:27:12,144][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:17,156][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:22,160][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:27,163][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:32,167][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:37,177][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:42,179][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:47,182][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:52,184][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:27:57,188][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:02,190][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:07,194][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:12,201][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:17,215][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:22,222][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:27,232][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:32,248][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:37,258][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:42,261][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:47,270][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:52,273][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:55,432][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4217, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147660_20260415_232459, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:28:55,432][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:28:55,434][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:28:56,496][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:28:56,496][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4217, hostname=2620:52:6:1161:dead:beef:cafe:c901, name=vmhost_p09_02_dev_02147660_20260415_232459, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:28:56,497][ INFO][PID:2893799] VM Release request [2026-04-15 23:28:56,508][ INFO][PID:2893799] Retry #9 (on other host) [2026-04-15 23:28:56,509][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:28:56,510][ INFO][PID:2893799] VM allocation process starts [2026-04-15 23:28:56,555][ INFO][PID:2893799] Trying to allocate VM: ResallocHost, ticket_id=4250, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:29:01,556][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:06,563][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:11,566][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:16,571][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:21,580][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:26,589][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:31,592][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:36,601][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:41,606][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:46,615][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:51,620][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:29:56,625][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:01,633][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:06,649][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:11,665][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:16,672][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:21,683][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:26,691][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:31,694][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:36,700][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:41,704][ INFO][PID:2893799] Checking for cancel request [2026-04-15 23:30:44,883][ INFO][PID:2893799] Allocated host ResallocHost, ticket_id=4250, hostname=2620:52:6:1161:dead:beef:cafe:c904, name=vmhost_x86_02_dev_02147673_20260415_232757, requested_tags=['arch_noarch', 'copr_builder'] [2026-04-15 23:30:44,884][ INFO][PID:2893799] Allocating ssh connection to builder [2026-04-15 23:30:44,885][ INFO][PID:2893799] Checking that builder machine is OK [2026-04-15 23:30:45,500][ INFO][PID:2893799] Installed copr-rpmbuild version: 1.6 [2026-04-15 23:30:45,501][ ERROR][PID:2893799] Re-try request for task on 'ResallocHost, ticket_id=4250, hostname=2620:52:6:1161:dead:beef:cafe:c904, name=vmhost_x86_02_dev_02147673_20260415_232757, requested_tags=['arch_noarch', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 23:30:45,502][ INFO][PID:2893799] VM Release request [2026-04-15 23:30:45,527][ ERROR][PID:2893799] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c904', '2620:52:6:1161:dead:beef:cafe:c901', '2620:52:6:1161:dead:beef:cafe:c906'} [2026-04-15 23:30:45,529][WARNING][PID:2893799] Switching not-finished job state to 'failed' [2026-04-15 23:30:45,530][ INFO][PID:2893799] Worker failed build, took 329.9683835506439 [2026-04-15 23:30:45,533][ INFO][PID:2893799] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.stg.fedoraproject.org", "memory_reqs": null, "enable_net": true, "project_owner": "@copr", "project_name": "TEST1776292405179016455-build-batches", "project_dirname": "TEST1776292405179016455-build-batches", "submitter": "frostyx", "ended_on": 1776295845.5298152, "started_on": 1776295515.5614316, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": null, "buildroot_pkgs": null, "task_id": "2925275", "build_id": 2925275, "package_name": "testpkg", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 8, "source_json": "{\"type\": \"git\", \"clone_url\": \"https://github.com/fedora-copr/copr-test-hello.git\", \"subdirectory\": \"\", \"committish\": \"\", \"spec\": \"\", \"srpm_build_method\": \"tito\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@copr/TEST1776292405179016455-build-batches--frostyx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [ "fedora-43" ], "distributions_in_project": [ "fedora-43" ], "destdir": "/var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches", "results_repo_url": "https://download.copr-dev.fedorainfracloud.org/results/@copr/TEST1776292405179016455-build-batches", "result_dir": "02925275", "built_packages": "", "tags": [ "arch_noarch" ], "id": 2925275, "mockchain_macros": { "copr_username": "@copr", "copr_projectname": "TEST1776292405179016455-build-batches", "vendor": "Fedora Project COPR (@copr/TEST1776292405179016455-build-batches)" } } ] } [2026-04-15 23:30:45,634][ INFO][PID:2893799] Sending fedora-messaging bus message in build.end [2026-04-15 23:30:47,794][ INFO][PID:2893799] Compressing /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/builder-live.log by gzip [2026-04-15 23:30:47,802][ INFO][PID:2893799] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/builder-live.log' as PID 2906882 [2026-04-15 23:30:47,807][ INFO][PID:2893799] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/builder-live.log: No such file or directory [2026-04-15 23:30:47,810][ ERROR][PID:2893799] Unable to compress file /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/builder-live.log [2026-04-15 23:30:47,815][ INFO][PID:2893799] Compressing /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/backend.log by gzip [2026-04-15 23:30:47,821][ INFO][PID:2893799] Running command 'gzip /var/lib/copr/public_html/results/@copr/TEST1776292405179016455-build-batches/srpm-builds/02925275/backend.log' as PID 2906883