[2026-04-15 22:26:42,209][ INFO][PID:2754879] Marking build as starting [2026-04-15 22:26:42,466][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:26:42,482][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:26:42,620][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2843, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:45,771][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2843, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:45,772][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:26:45,776][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:26:46,521][ INFO][PID:2754879] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:26:46,522][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2843, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:26:46,524][ INFO][PID:2754879] VM Release request [2026-04-15 22:26:46,549][ INFO][PID:2754879] Retry #1 (on other host) [2026-04-15 22:26:46,550][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:26:46,558][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:26:46,636][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2852, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:51,642][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:26:52,774][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2852, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:52,777][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:26:52,782][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:26:52,923][ INFO][PID:2754879] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:26:52,928][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2852, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:26:52,930][ INFO][PID:2754879] VM Release request [2026-04-15 22:26:52,963][ INFO][PID:2754879] Retry #2 (on other host) [2026-04-15 22:26:52,964][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:26:52,973][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:26:52,993][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2853, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:56,062][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2853, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:26:56,063][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:26:56,069][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:26:56,195][ INFO][PID:2754879] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:26:56,196][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2853, hostname=2620:52:6:1161:dead:beef:cafe:c906, name=vmhost_x86_01_dev_02147434_20260415_214128, requested_tags=['arch_x86_64', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:26:56,197][ INFO][PID:2754879] VM Release request [2026-04-15 22:26:56,247][ INFO][PID:2754879] Retry #3 (on other host) [2026-04-15 22:26:56,250][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:26:56,257][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:26:56,320][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2854, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:27:01,336][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:06,348][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:11,351][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:16,352][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:21,357][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:26,360][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:31,364][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:36,366][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:41,377][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:46,382][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:51,384][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:27:56,385][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:01,392][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:06,394][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:11,405][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:16,406][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:21,412][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:26,418][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:31,422][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:36,424][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:41,427][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:46,430][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:51,437][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:28:56,442][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:01,450][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:06,460][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:11,463][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:14,722][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2854, hostname=2620:52:6:1161:dead:beef:cafe:c90a, name=vmhost_x86_03_dev_02147437_20260415_222644, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:29:14,726][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:29:14,728][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:29:15,379][ INFO][PID:2754879] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:29:15,381][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2854, hostname=2620:52:6:1161:dead:beef:cafe:c90a, name=vmhost_x86_03_dev_02147437_20260415_222644, requested_tags=['arch_x86_64', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:29:15,387][ INFO][PID:2754879] VM Release request [2026-04-15 22:29:15,431][ INFO][PID:2754879] Retry #4 (on other host) [2026-04-15 22:29:15,436][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:15,444][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:29:15,495][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2867, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:29:20,502][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:21,602][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2867, hostname=2620:52:6:1161:dead:beef:cafe:c90a, name=vmhost_x86_03_dev_02147437_20260415_222644, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:29:21,603][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:29:21,604][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:29:21,747][ INFO][PID:2754879] Installed copr-rpmbuild version: 1.6 [2026-04-15 22:29:21,749][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2867, hostname=2620:52:6:1161:dead:beef:cafe:c90a, name=vmhost_x86_03_dev_02147437_20260415_222644, requested_tags=['arch_x86_64', 'copr_builder']': Minimum version for builder is 1.6.1 [2026-04-15 22:29:21,751][ INFO][PID:2754879] VM Release request [2026-04-15 22:29:21,790][ INFO][PID:2754879] Retry #5 (on other host) [2026-04-15 22:29:21,792][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:21,800][ INFO][PID:2754879] VM allocation process starts [2026-04-15 22:29:21,900][ INFO][PID:2754879] Trying to allocate VM: ResallocHost, ticket_id=2868, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:29:26,906][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:31,912][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:36,918][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:41,922][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:46,923][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:51,927][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:29:56,929][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:01,930][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:06,934][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:11,936][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:16,939][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:21,945][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:26,947][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:31,950][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:36,953][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:41,958][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:46,962][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:51,966][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:30:56,974][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:01,980][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:06,982][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:11,986][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:16,988][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:21,991][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:26,995][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:31,997][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:36,998][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:42,000][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:47,003][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:52,019][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:31:57,026][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:02,028][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:07,033][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:12,035][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:17,038][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:22,044][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:27,046][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:32,050][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:37,056][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:42,058][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:47,064][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:52,069][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:32:57,071][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:02,072][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:07,081][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:12,084][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:17,088][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:22,096][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:27,100][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:32,102][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:37,114][ INFO][PID:2754879] Checking for cancel request [2026-04-15 22:33:40,207][ INFO][PID:2754879] Allocated host ResallocHost, ticket_id=2868, hostname=2620:52:6:1161:dead:beef:cafe:c905, name=vmhost_x86_03_dev_02147451_20260415_223110, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-15 22:33:40,208][ INFO][PID:2754879] Allocating ssh connection to builder [2026-04-15 22:33:40,210][ INFO][PID:2754879] Checking that builder machine is OK [2026-04-15 22:34:40,316][ ERROR][PID:2754879] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c905 port 22: Connection timed out [2026-04-15 22:34:50,321][ ERROR][PID:2754879] Re-try request for task on 'ResallocHost, ticket_id=2868, hostname=2620:52:6:1161:dead:beef:cafe:c905, name=vmhost_x86_03_dev_02147451_20260415_223110, requested_tags=['arch_x86_64', 'copr_builder']': Unable to finish after 1 SSH attempts [2026-04-15 22:34:50,325][ INFO][PID:2754879] VM Release request [2026-04-15 22:34:50,390][ ERROR][PID:2754879] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c90a', '2620:52:6:1161:dead:beef:cafe:c905', '2620:52:6:1161:dead:beef:cafe:c906'} [2026-04-15 22:34:50,398][WARNING][PID:2754879] Switching not-finished job state to 'failed' [2026-04-15 22:34:50,410][ INFO][PID:2754879] Worker failed build, took 488.2202591896057 [2026-04-15 22:34:50,428][ INFO][PID:2754879] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.stg.fedoraproject.org", "memory_reqs": null, "enable_net": true, "project_owner": "frostyx", "project_name": "custom-1-TEST1776291979831740462", "project_dirname": "custom-1-TEST1776291979831740462", "submitter": "frostyx", "ended_on": 1776292490.3987541, "started_on": 1776292002.178495, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "2925225", "build_id": 2925225, "package_name": "quick-package", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#! /bin/sh -x\\n\\nset -e\\n\\ngenerate_specfile()\\n{\\n test -n \\\"$DESTDIR\\\" && mkdir -p \\\"$DESTDIR\\\"\\n\\n test -n \\\"$BUILDDEPS\\\" && {\\n for i in $BUILDDEPS; do\\n rpm -q $i\\n done\\n }\\n\\n if ${HOOK_PAYLOAD-false}; then\\n test -f hook_payload\\n test \\\"$(cat hook_payload)\\\" = \\\"{\\\\\\\"a\\\\\\\": \\\\\\\"b\\\\\\\"}\\\"\\n else\\n ! test -f hook_payload\\n fi\\n\\ncat > \\\"${DESTDIR-.}\\\"/quick-package.spec <<\\\\EOF\\nName: quick-package\\nVersion: 0\\nRelease: 0%{?dist}\\nSummary: dummy package\\nLicense: GPL\\nURL: http://example.com/\\n\\n%{!?_pkgdocdir: %global _pkgdocdir %{_docdir}/%{name}-%{version}}\\n\\n%description\\nnothing\\n\\n\\n%install\\nmkdir -p $RPM_BUILD_ROOT/%{_pkgdocdir}\\necho \\\"this does nothing\\\" > $RPM_BUILD_ROOT/%{_pkgdocdir}/README\\n\\n\\n%files\\n%doc %{_pkgdocdir}/README\\n\\n\\n%changelog\\n* Thu Jun 05 2014 Pavel Raiskup - 0-1\\n- does nothing!\\nEOF\\n}\\ngenerate_specfile\\n\", \"chroot\": \"fedora-rawhide-x86_64\", \"builddeps\": null, \"resultdir\": null, \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "frostyx/custom-1-TEST1776291979831740462--frostyx", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [ "fedora-rawhide" ], "distributions_in_project": [ "fedora-rawhide" ], "destdir": "/var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462", "results_repo_url": "https://download.copr-dev.fedorainfracloud.org/results/frostyx/custom-1-TEST1776291979831740462", "result_dir": "02925225", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 2925225, "mockchain_macros": { "copr_username": "frostyx", "copr_projectname": "custom-1-TEST1776291979831740462", "vendor": "Fedora Project COPR (frostyx/custom-1-TEST1776291979831740462)" } } ] } [2026-04-15 22:34:51,220][ INFO][PID:2754879] Sending fedora-messaging bus message in build.end [2026-04-15 22:35:01,497][ INFO][PID:2754879] Compressing /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/builder-live.log by gzip [2026-04-15 22:35:01,509][ INFO][PID:2754879] Running command 'gzip /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/builder-live.log' as PID 2771671 [2026-04-15 22:35:01,521][ INFO][PID:2754879] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/builder-live.log) stdout: stderr: gzip: /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/builder-live.log: No such file or directory [2026-04-15 22:35:01,523][ ERROR][PID:2754879] Unable to compress file /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/builder-live.log [2026-04-15 22:35:01,532][ INFO][PID:2754879] Compressing /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/backend.log by gzip [2026-04-15 22:35:01,543][ INFO][PID:2754879] Running command 'gzip /var/lib/copr/public_html/results/frostyx/custom-1-TEST1776291979831740462/srpm-builds/02925225/backend.log' as PID 2771672