run perforator across dozens of clusters #34

soanni86 · 2025-02-10T21:53:10Z

soanni86
Feb 10, 2025

Hello dear maintainers,

please advise i want to profile my applications across dozens (hundreds) of k8s clusters. As far as i understood i need to run only agents on those clusters and all other microservices i can run on separate cluster right ?

for agent i only need to provide a storage service url + grpc port so in my case i will probably make it type loadbalancer + grpc.

also i want to introduce a label 'cluster_name' and use it in queries but as far as i can read from the code the labels are pre-defined and possibly the candidate is to use topologyLableKey and put there (instead of a zone as for me zone doesnt make any diff tbh) the node label that contains cluster name the node belongs to ?
is there a way to introduce custom labels ? in highly distributed environments that would be really nice (possible candidates: env (dev/prod/stage), dc (america, africa, europe, australia, etc), cluster_name) ?

thanks

MikailBag · 2025-02-11T08:08:55Z

MikailBag
Feb 11, 2025
Maintainer

Hello.

As far as i understood i need to run only agents on those clusters and all other microservices i can run on separate cluster right

You are right. Single Perforator installation can have agents running on multiple Kubernetes clusters. Native support for this scenario in chart is tracked in #19. In fact, starting with v0.2.3 there is already some support, but I'm not sure whether it is enough for your case.

is there a way to introduce custom labels?

There is support for it in the agent. Unfortunately, there are no easy ways to specify this field when using chart. I have created #35 to track this feature request.

is to use topologyLableKey and put there

If you don't need to distinguish nodes within one cluster (based on rack / AZ / DC / etc), then this workaround should work.

0 replies

soanni86 · 2025-02-11T21:58:18Z

soanni86
Feb 11, 2025
Author

was trying today to deploy agents only on one of my clusters and make them talk to "central" cluster that runs storage/web/proxy/gc/offline/agents, i created small PR https://github.com/yandex/perforator/pull/36/files
So, to be able to reach storage service i made it load balancer and assigned a dns to it (domain is in internal zone)

storage:
  service:
    type: LoadBalancer
    annotations:
      networking.gke.io/load-balancer-type: Internal
      networking.gke.io/internal-load-balancer-allow-global-access: "true"
      external-dns.alpha.kubernetes.io/hostname: perforator-storage-grpc.dev-corp.com

then added hostname

hostname: "perforator-storage-grpc.dev-corp.com:7618"

i also issued a cert and put it to tls secret k8s obj for perforator-storage-grpc.dev-corp.com via acme so not to use a self-signed for storage server

storageAgentTLS:
  autoGenerated: false
  storage:
    existingSecret: perforator-storage-grpc-issued-tls
    certFilename: "tls.crt"
    certKeyFilename: "tls.key"
    certCAFilename: "ca.crt"

central cluster agents work fine and talk to storage via LB and use the cert, but agents (i use latest version v0.0.3) on another cluster crashloop, some with some huge stack trace

/home/runner/work/perforator/perforator/contrib/libs/tcmalloc/tcmalloc/internal/percpu_tcmalloc.h:1030] !hdr.IsLocked() @
SIGABRT: abort
PC=0x2acb2c7 m=9 sigcode=18446744073709551610

goroutine 0 gp=0xc0006221c0 m=9 mp=0xc000620000 [idle]:
runtime: g 0 gp=0xc0006221c0: unknown pc 0x2acb2c7
stack: frame={sp:0x7989f76c28a8, fp:0x0} stack=[0x7989f6ef3ca0,0x7989f76c38a0)
0x00007989f76c27a8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27b8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27c8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27d8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27e8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27f8:  0x0000000000000000  0x0000000000000000
0x00007989f76c2808:  0x0000000000000000  0x0000000000000000
0x00007989f76c2818:  0x0000000000000000  0x0000000000000000
0x00007989f76c2828:  0x0000000000000000  0x0000000000000000
0x00007989f76c2838:  0x0000000000000000  0x0000000000000000
0x00007989f76c2848:  0x0000000000000000  0x0000000000000000
0x00007989f76c2858:  0x0000000000000000  0x0000000000000000
0x00007989f76c2868:  0x0000000000000000  0x0000000000000000
0x00007989f76c2878:  0x0000000000000000  0x0000000000000000
0x00007989f76c2888:  0x0000000000000000  0x0000000000000000
0x00007989f76c2898:  0x0000000000000000  0x0000000000000000
0x00007989f76c28a8: <0x0000000002acb313  0x0000000000000000
0x00007989f76c28b8:  0x0000000000000000  0x0000000000000000
0x00007989f76c28c8:  0x0000000000000000  0x0000000000000000
0x00007989f76c28d8:  0x0000000002a5c572  0x0000000000000040
0x00007989f76c28e8:  0x000000000431588b  0x00007989f76c2990
0x00007989f76c28f8:  0x00007989f76c2a3e  0x00007989f76c29b0
0x00007989f76c2908:  0x00007989f76c29c8  0x00007989f76c2990
0x00007989f76c2918:  0x00007989f76c29b0  0x00007989f76c29a0
0x00007989f76c2928:  0x0000000004315429  0x0000000000000000
0x00007989f76c2938:  0x0000000000000000  0x0000000000000000
0x00007989f76c2948:  0x0000000002ac13ee  0x0000000000c3e02a
0x00007989f76c2958:  0x0000000000000000  0x00000000042f7340
0x00007989f76c2968:  0x0000ffff00000000  0x0000000004575b80
0x00007989f76c2978:  0x0000000002adca22  0x0000000000000000
0x00007989f76c2988:  0x00000000043155b6  0x0000000000000004
0x00007989f76c2998:  0x0000000000000000  0x0000000000000004
runtime: g 0 gp=0xc0006221c0: unknown pc 0x2acb2c7
stack: frame={sp:0x7989f76c28a8, fp:0x0} stack=[0x7989f6ef3ca0,0x7989f76c38a0)
0x00007989f76c27a8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27b8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27c8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27d8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27e8:  0x0000000000000000  0x0000000000000000
0x00007989f76c27f8:  0x0000000000000000  0x0000000000000000
0x00007989f76c2808:  0x0000000000000000  0x0000000000000000
0x00007989f76c2818:  0x0000000000000000  0x0000000000000000
0x00007989f76c2828:  0x0000000000000000  0x0000000000000000
0x00007989f76c2838:  0x0000000000000000  0x0000000000000000
0x00007989f76c2848:  0x0000000000000000  0x0000000000000000
0x00007989f76c2858:  0x0000000000000000  0x0000000000000000
0x00007989f76c2868:  0x0000000000000000  0x0000000000000000
0x00007989f76c2878:  0x0000000000000000  0x0000000000000000
0x00007989f76c2888:  0x0000000000000000  0x0000000000000000
0x00007989f76c2898:  0x0000000000000000  0x0000000000000000
0x00007989f76c28a8: <0x0000000002acb313  0x0000000000000000
0x00007989f76c28b8:  0x0000000000000000  0x0000000000000000
0x00007989f76c28c8:  0x0000000000000000  0x0000000000000000
0x00007989f76c28d8:  0x0000000002a5c572  0x0000000000000040
0x00007989f76c28e8:  0x000000000431588b  0x00007989f76c2990
0x00007989f76c28f8:  0x00007989f76c2a3e  0x00007989f76c29b0
0x00007989f76c2908:  0x00007989f76c29c8  0x00007989f76c2990
0x00007989f76c2918:  0x00007989f76c29b0  0x00007989f76c29a0
0x00007989f76c2928:  0x0000000004315429  0x0000000000000000
0x00007989f76c2938:  0x0000000000000000  0x0000000000000000
0x00007989f76c2948:  0x0000000002ac13ee  0x0000000000c3e02a
0x00007989f76c2958:  0x0000000000000000  0x00000000042f7340
0x00007989f76c2968:  0x0000ffff00000000  0x0000000004575b80
0x00007989f76c2978:  0x0000000002adca22  0x0000000000000000
0x00007989f76c2988:  0x00000000043155b6  0x0000000000000004
0x00007989f76c2998:  0x0000000000000000  0x0000000000000004

goroutine 1 gp=0xc0000061c0 m=nil [runnable]:
bufio.(*Scanner).Scan(0xc0009f4088)
	/-S/contrib/go/_std_1.22/src/bufio/scan.go:139 +0x8c5 fp=0xc0009f3fc8 sp=0xc0009f3fc0 pc=0x2091da5
github.com/cilium/ebpf/btf.readStringTable({0xa7cce8, 0xc000819530}, 0x0)
	/-S/vendor/github.com/cilium/ebpf/btf/strings.go:46 +0x1c5 fp=0xc0009f4118 sp=0xc0009f3fc8 pc=0x238c365
github.com/cilium/ebpf/btf.parseBTF({0xa768a0, 0xc0003060d0}, {0xa89320, 0x4ea4240}, 0x0, 0x0)
	/-S/vendor/github.com/cilium/ebpf/btf/btf.go:412 +0x198 fp=0xc0009f4210 sp=0xc0009f4118 pc=0x2377538
github.com/cilium/ebpf/btf.loadRawSpec({0xa768a0?, 0xc0003060d0?}, {0xa89320, 0x4ea4240}, 0x200?)
	/-S/vendor/github.com/cilium/ebpf/btf/btf.go:338 +0x105 fp=0xc0009f42a8 sp=0xc0009f4210 pc=0x2376cc5
github.com/cilium/ebpf/btf.loadKernelSpec()
	/-S/vendor/github.com/cilium/ebpf/btf/kernel.go:100 +0x145 fp=0xc0009f4330 sp=0xc0009f42a8 pc=0x23881e5
github.com/cilium/ebpf/btf.LoadKernelSpec()
	/-S/vendor/github.com/cilium/ebpf/btf/kernel.go:51 +0x187 fp=0xc0009f43a0 sp=0xc0009f4330 pc=0x2387a67
github.com/cilium/ebpf.applyRelocations({0xc0008e6000?, 0x4?, 0x714b10399320a735?}, {0x0, 0x0, 0x0}, {0x0, 0x0}, {0xa89320, 0x4ea4240}, ...)
	/-S/vendor/github.com/cilium/ebpf/linker.go:146 +0x2e9 fp=0xc0009f4540 sp=0xc0009f43a0 pc=0x23b4fc9
github.com/cilium/ebpf.newProgramWithOptions(0xc0009f4aa8, {0x0, 0x600000, 0x0, 0x0, 0x0})
	/-S/vendor/github.com/cilium/ebpf/prog.go:314 +0x645 fp=0xc0009f49b0 sp=0xc0009f4540 pc=0x23c13c5
github.com/cilium/ebpf.(*collectionLoader).loadProgram(0xc0007c7590, {0x2abfe6, 0x1d})
	/-S/vendor/github.com/cilium/ebpf/collection.go:577 +0x42e fp=0xc0009f4bd8 sp=0xc0009f49b0 pc=0x23a532e
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign.func1({0xa95910, 0x4182c0}, {0x2abfe6, 0x1d})
	/-S/vendor/github.com/cilium/ebpf/collection.go:271 +0x2e5 fp=0xc0009f4c70 sp=0xc0009f4bd8 pc=0x23a4185
github.com/cilium/ebpf.assignValues({0x31f920, 0xc00025d620}, 0xc0009f5078)
	/-S/vendor/github.com/cilium/ebpf/collection.go:1022 +0x412 fp=0xc0009f5020 sp=0xc0009f4c70 pc=0x23a7db2
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign(0xc0007c6a50, {0x31f920, 0xc00025d620}, 0x0?)
	/-S/vendor/github.com/cilium/ebpf/collection.go:287 +0x21b fp=0xc0009f5390 sp=0xc0009f5020 pc=0x23a3abb
github.com/yandex/perforator/perforator/agent/collector/pkg/machine.(*BPF).setupProgramsUnsafe(0xc00072a000, 0x0)
	/-S/perforator/agent/collector/pkg/machine/bpf.go:313 +0x1cd fp=0xc0009f5570 sp=0xc0009f5390 pc=0x23de10d
github.com/yandex/perforator/perforator/agent/collector/pkg/machine.(*BPF).initialize(0xc00072a000)
	/-S/perforator/agent/collector/pkg/machine/bpf.go:157 +0x1cb fp=0xc0009f55d8 sp=0xc0009f5570 pc=0x23dd30b
github.com/yandex/perforator/perforator/agent/collector/pkg/machine.NewBPF(0xc0005147e8, {0xa8dcd0, 0xc000306590}, {0xa8f1e0?, 0xc000322aa0?})
	/-S/perforator/agent/collector/pkg/machine/bpf.go:123 +0x29c fp=0xc0009f5680 sp=0xc0009f55d8 pc=0x23dd0bc
github.com/yandex/perforator/perforator/agent/collector/pkg/profiler.(*Profiler).initialize(0xc00058c200, {0xa8f1e0, 0xc000322aa0})
	/-S/perforator/agent/collector/pkg/profiler/profiler.go:217 +0x5a fp=0xc0009f5870 sp=0xc0009f5680 pc=0x2a170fa
github.com/yandex/perforator/perforator/agent/collector/pkg/profiler.NewProfiler(0xc0005147e0, {0xa8dcd0, 0xc000306510}, {0xa8f1e0, 0xc000322aa0}, {0x0, 0x0, 0x0?})
	/-S/perforator/agent/collector/pkg/profiler/profiler.go:182 +0x5ca fp=0xc0009f5a08 sp=0xc0009f5870 pc=0x2a16dea
main.run()
	/-S/perforator/cmd/agent/main.go:161 +0x677 fp=0xc0009f5c88 sp=0xc0009f5a08 pc=0x2a26c37
main.init.func1(0xc000250300?, {0x485a3a?, 0x4?, 0x485a3e?})
	/-S/perforator/cmd/agent/main.go:44 +0xf fp=0xc0009f5c98 sp=0xc0009f5c88 pc=0x2a25fef
github.com/spf13/cobra.(*Command).execute(0x4561960, {0xc0001a0190, 0x2, 0x2})
	/-S/vendor/github.com/spf13/cobra/command.go:985 +0xaca fp=0xc0009f5e20 sp=0xc0009f5c98 pc=0x21cf5ea
github.com/spf13/cobra.(*Command).ExecuteC(0x4561960)
	/-S/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff fp=0xc0009f5ef8 sp=0xc0009f5e20 pc=0x21cfebf
github.com/spf13/cobra.(*Command).Execute(...)
	/-S/vendor/github.com/spf13/cobra/command.go:1041
main.main()
	/-S/perforator/cmd/agent/main.go:75 +0x1f fp=0xc0009f5f50 sp=0xc0009f5ef8 pc=0x2a2631f
runtime.main()
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:271 +0x29d fp=0xc0009f5fe0 sp=0xc0009f5f50 pc=0x1e56b1d
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0009f5fe8 sp=0xc0009f5fe0 pc=0x1e8a021

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000088fa8 sp=0xc000088f88 pc=0x1e56f6e
runtime.goparkunlock(...)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:408
runtime.forcegchelper()
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:326 +0xb3 fp=0xc000088fe0 sp=0xc000088fa8 pc=0x1e56dd3
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000088fe8 sp=0xc000088fe0 pc=0x1e8a021
created by runtime.init.6 in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000089780 sp=0xc000089760 pc=0x1e56f6e
runtime.goparkunlock(...)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:408
runtime.bgsweep(0xc0000b2000)
	/-S/contrib/go/_std_1.22/src/runtime/mgcsweep.go:318 +0xdf fp=0xc0000897c8 sp=0xc000089780 pc=0x1e409ff
runtime.gcenable.gowrap1()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:203 +0x25 fp=0xc0000897e0 sp=0xc0000897c8 pc=0x1e355c5
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000897e8 sp=0xc0000897e0 pc=0x1e8a021
created by runtime.gcenable in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0xa6dc60?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000089f78 sp=0xc000089f58 pc=0x1e56f6e
runtime.goparkunlock(...)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x45844a0)
	/-S/contrib/go/_std_1.22/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000089fa8 sp=0xc000089f78 pc=0x1e3e3a9
runtime.bgscavenge(0xc0000b2000)
	/-S/contrib/go/_std_1.22/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000089fc8 sp=0xc000089fa8 pc=0x1e3e959
runtime.gcenable.gowrap2()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:204 +0x25 fp=0xc000089fe0 sp=0xc000089fc8 pc=0x1e35565
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000089fe8 sp=0xc000089fe0 pc=0x1e8a021
created by runtime.gcenable in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc000186380 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x5d5980?, 0x40?, 0x1?, 0x2000000020?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000088620 sp=0xc000088600 pc=0x1e56f6e
runtime.runfinq()
	/-S/contrib/go/_std_1.22/src/runtime/mfinal.go:194 +0x107 fp=0xc0000887e0 sp=0xc000088620 pc=0x1e345a7
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000887e8 sp=0xc0000887e0 pc=0x1e8a021
created by runtime.createfing in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc0002276c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000084750 sp=0xc000084730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000847e0 sp=0xc000084750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 5 gp=0xc000007880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc00008a750 sp=0xc00008a730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008a7e0 sp=0xc00008a750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc000582000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000588750 sp=0xc000588730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005887e0 sp=0xc000588750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005887e8 sp=0xc0005887e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 6 gp=0xc000007a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc00008af50 sp=0xc00008af30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008afe0 sp=0xc00008af50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc000227880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000084f50 sp=0xc000084f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000084fe0 sp=0xc000084f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc0005821c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000588f50 sp=0xc000588f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000588fe0 sp=0xc000588f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 36 gp=0xc000582380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000589750 sp=0xc000589730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005897e0 sp=0xc000589750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 37 gp=0xc000582540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000589f50 sp=0xc000589f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000589fe0 sp=0xc000589f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000589fe8 sp=0xc000589fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc000227a40 m=nil [runnable]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000085750 sp=0xc000085730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000857e0 sp=0xc000085750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 7 gp=0xc000007c00 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x1?, 0x4a?, 0x22?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc00008b750 sp=0xc00008b730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008b7e0 sp=0xc00008b750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 22 gp=0xc000227c00 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x3?, 0xf2?, 0x73?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000085f50 sp=0xc000085f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000085fe0 sp=0xc000085f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc000007dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x1?, 0xc4?, 0x63?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc00008bf50 sp=0xc00008bf30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 23 gp=0xc000227dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x1?, 0xe0?, 0xd8?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000086750 sp=0xc000086730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000867e0 sp=0xc000086750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc0000c0000 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x3?, 0xc2?, 0x80?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000584750 sp=0xc000584730 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005847e0 sp=0xc000584750 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005847e8 sp=0xc0005847e0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 24 gp=0xc00051a000 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x3?, 0xae?, 0x8b?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000086f50 sp=0xc000086f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000086fe0 sp=0xc000086f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 10 gp=0xc0000c01c0 m=nil [GC worker (idle)]:
runtime.gopark(0x4ea5fc0?, 0x1?, 0xc0?, 0xe1?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc000584f50 sp=0xc000584f30 pc=0x1e56f6e
runtime.gcBgMarkWorker()
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1310 +0xe5 fp=0xc000584fe0 sp=0xc000584f50 pc=0x1e37645
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000584fe8 sp=0xc000584fe0 pc=0x1e8a021
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/-S/contrib/go/_std_1.22/src/runtime/mgc.go:1234 +0x1c

goroutine 25 gp=0xc0000c1500 m=nil [select]:
runtime.gopark(0xc000587768?, 0x2?, 0x0?, 0x66?, 0xc00058772c?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc0005875d0 sp=0xc0005875b0 pc=0x1e56f6e
runtime.selectgo(0xc000587768, 0xc000587728, 0x0?, 0x0, 0xc0001ad9e0?, 0x1)
	/-S/contrib/go/_std_1.22/src/runtime/select.go:327 +0x725 fp=0xc0005876f0 sp=0xc0005875d0 pc=0x1e67fa5
github.com/yandex/perforator/library/go/core/log/zap/asynczap.(*background).run(0xc000232300)
	/-S/library/go/core/log/zap/asynczap/background.go:113 +0x1f9 fp=0xc0005877c8 sp=0xc0005876f0 pc=0x2272599
github.com/yandex/perforator/library/go/core/log/zap/asynczap.NewCore.gowrap1()
	/-S/library/go/core/log/zap/asynczap/core.go:43 +0x25 fp=0xc0005877e0 sp=0xc0005877c8 pc=0x2272c65
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005877e8 sp=0xc0005877e0 pc=0x1e8a021
created by github.com/yandex/perforator/library/go/core/log/zap/asynczap.NewCore in goroutine 1
	/-S/library/go/core/log/zap/asynczap/core.go:43 +0x1d6

goroutine 14 gp=0xc000103500 m=nil [IO wait]:
runtime.gopark(0xc0000d3718?, 0x1e2e5bf?, 0xc0?, 0xbb?, 0xb?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc0000d36f0 sp=0xc0000d36d0 pc=0x1e56f6e
runtime.netpollblock(0x1ea73f8?, 0x1e1d866?, 0x0?)
	/-S/contrib/go/_std_1.22/src/runtime/netpoll.go:573 +0xf7 fp=0xc0000d3728 sp=0xc0000d36f0 pc=0x1e4fb57
internal/poll.runtime_pollWait(0x7989f8ff5e40, 0x72)
	/-S/contrib/go/_std_1.22/src/runtime/netpoll.go:345 +0x85 fp=0xc0000d3748 sp=0xc0000d3728 pc=0x1e84a65
internal/poll.(*pollDesc).wait(0xc000127000?, 0xc0006be000?, 0x0)
	/-S/contrib/go/_std_1.22/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000d3770 sp=0xc0000d3748 pc=0x1efb847
internal/poll.(*pollDesc).waitRead(...)
	/-S/contrib/go/_std_1.22/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000127000, {0xc0006be000, 0x3800, 0x3800})
	/-S/contrib/go/_std_1.22/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0000d3808 sp=0xc0000d3770 pc=0x1efcb3a
net.(*netFD).Read(0xc000127000, {0xc0006be000?, 0x49528d?, 0x10?})
	/-S/contrib/go/_std_1.22/src/net/fd_posix.go:55 +0x25 fp=0xc0000d3850 sp=0xc0000d3808 pc=0x1fe8965
net.(*conn).Read(0xc000110068, {0xc0006be000?, 0x6?, 0x485942?})
	/-S/contrib/go/_std_1.22/src/net/net.go:185 +0x45 fp=0xc0000d3898 sp=0xc0000d3850 pc=0x1ff9005
net.(*TCPConn).Read(0x1e27b17?, {0xc0006be000?, 0x18?, 0x18?})
	<autogenerated>:1 +0x25 fp=0xc0000d38c8 sp=0xc0000d3898 pc=0x200a6c5
crypto/tls.(*atLeastReader).Read(0xc0000125b8, {0xc0006be000?, 0xc0000125b8?, 0x0?})
	/-S/contrib/go/_std_1.22/src/crypto/tls/conn.go:806 +0x3b fp=0xc0000d3910 sp=0xc0000d38c8 pc=0x204e6fb
bytes.(*Buffer).ReadFrom(0xc0000c29a8, {0xa76740, 0xc0000125b8})
	/-S/contrib/go/_std_1.22/src/bytes/buffer.go:211 +0x98 fp=0xc0000d3968 sp=0xc0000d3910 pc=0x1e9e798
crypto/tls.(*Conn).readFromUntil(0xc0000c2700, {0xa769e0, 0xc000110068}, 0xc0000d3978?)
	/-S/contrib/go/_std_1.22/src/crypto/tls/conn.go:828 +0xde fp=0xc0000d39a0 sp=0xc0000d3968 pc=0x204e8de
crypto/tls.(*Conn).readRecordOrCCS(0xc0000c2700, 0x0)
	/-S/contrib/go/_std_1.22/src/crypto/tls/conn.go:626 +0x3cf fp=0xc0000d3c20 sp=0xc0000d39a0 pc=0x204b9ef
crypto/tls.(*Conn).readRecord(...)
	/-S/contrib/go/_std_1.22/src/crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0xc0000c2700, {0xc00063c000, 0x1000, 0x8?})
	/-S/contrib/go/_std_1.22/src/crypto/tls/conn.go:1370 +0x156 fp=0xc0000d3c90 sp=0xc0000d3c20 pc=0x2052296
net/http.(*persistConn).Read(0xc000514c60, {0xc00063c000?, 0xc0000d42a0?, 0xc0000d3d38?})
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:1977 +0x4a fp=0xc0000d3cf0 sp=0xc0000d3c90 pc=0x21274ca
bufio.(*Reader).fill(0xc0000e1a40)
	/-S/contrib/go/_std_1.22/src/bufio/bufio.go:110 +0x103 fp=0xc0000d3d28 sp=0xc0000d3cf0 pc=0x208eca3
bufio.(*Reader).Peek(0xc0000e1a40, 0x1)
	/-S/contrib/go/_std_1.22/src/bufio/bufio.go:148 +0x53 fp=0xc0000d3d48 sp=0xc0000d3d28 pc=0x208edd3
net/http.(*persistConn).readLoop(0xc000514c60)
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:2141 +0x1b9 fp=0xc0000d3fc8 sp=0xc0000d3d48 pc=0x2128259
net/http.(*Transport).dialConn.gowrap2()
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:1799 +0x25 fp=0xc0000d3fe0 sp=0xc0000d3fc8 pc=0x21269e5
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000d3fe8 sp=0xc0000d3fe0 pc=0x1e8a021
created by net/http.(*Transport).dialConn in goroutine 27
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:1799 +0x152f

goroutine 15 gp=0xc00051a540 m=nil [select]:
runtime.gopark(0xc0000cff48?, 0x2?, 0xc0?, 0xfd?, 0xc0000cfef4?)
	/-S/contrib/go/_std_1.22/src/runtime/proc.go:402 +0xce fp=0xc0000cfd98 sp=0xc0000cfd78 pc=0x1e56f6e
runtime.selectgo(0xc0000cff48, 0xc0000cfef0, 0xc0000c8140?, 0x0, 0xc0000d8450?, 0x1)
	/-S/contrib/go/_std_1.22/src/runtime/select.go:327 +0x725 fp=0xc0000cfeb8 sp=0xc0000cfd98 pc=0x1e67fa5
net/http.(*persistConn).writeLoop(0xc000514c60)
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:2458 +0xf0 fp=0xc0000cffc8 sp=0xc0000cfeb8 pc=0x2129d50
net/http.(*Transport).dialConn.gowrap3()
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:1800 +0x25 fp=0xc0000cffe0 sp=0xc0000cffc8 pc=0x2126985
runtime.goexit({})
	/-S/contrib/go/_std_1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x1e8a021
created by net/http.(*Transport).dialConn in goroutine 27
	/-S/contrib/go/_std_1.22/src/net/http/transport.go:1800 +0x1585

rax    0x0
rbx    0x0
rcx    0x2acb2c7
rdx    0x0
rdi    0x2
rsi    0x7989f76c28b8
rbp    0x7a
rsp    0x7989f76c28a8
r8     0x0
r9     0x0
r10    0x8
r11    0x246
r12    0x42f7340
r13    0xffff00000000
r14    0x7989f76c28b8
r15    0x4
rip    0x2acb2c7
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

the Running pod also doesnt talk to storage and says cert authority is unknown

tskv	ts=2025-02-11T21:52:56.217008476Z	level=error	logger=profiler.storage.Client	available_build_ids=[8AtRBocAoGQNTic2NWu1/CSQQ_YXfGXPXEAMcRLeO/Sl0HuIEwVHIjvChBNHYa/jX211CNbwgWgRH72zqur]	msg=Failed announce binaries
tskv	ts=2025-02-11T21:52:56.217016536Z	level=error	logger=profiler.ProcessRegistry.uploader	msg=Failed to upload closed binary	build_id=8AtRBocAoGQNTic2NWu1/CSQQ_YXfGXPXEAMcRLeO/Sl0HuIEwVHIjvChBNHYa/jX211CNbwgWgRH72zqurerror=rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

see

k -nperforator get po
NAME                     READY   STATUS             RESTARTS        AGE
perforator-agent-56j8w   0/1     CrashLoopBackOff   9 (4m51s ago)   25m
perforator-agent-99wbj   1/1     Running            2 (3h1m ago)    3h1m
perforator-agent-j8xww   0/1     CrashLoopBackOff   9 (4m19s ago)   25m
perforator-agent-k5fsw   0/1     Pending            0               25m
perforator-agent-rmrbh   0/1     Pending            0               25m
perforator-agent-vnwsd   0/1     Pending            0               25m

i shared the same cert file to standalone cluster via extraDeploy object in my PR so the tls.ctr, ca.crt and tls.key are the same for both clusters.

extraDeploy:
  - |
    apiVersion: v1
    data:
      ca.crt: aaabbbccc==
      tls.crt: dddeeefff
      tls.key: xxxyyyzzz=
    kind: Secret
    metadata:
      name: perforator-storage-grpc-issued-tls
      namespace: perforator
    type: kubernetes.io/tls

i tried with self signed cert also and magic one pod was running and sending data just fine but others faced the same error as above. as far as i can see from code TLS is mandatory for server side so no option to test it w/o TLS

2 replies

MikailBag Feb 12, 2025
Maintainer

So, if I understood correctly, you tried two different configurations:

Storage certificate is issued by some private CA, ca.crt contains CA certificate
Storage certificate is self-signed, ca.crt is equal to tls.crt and contains that self-signed certificate.

Where in both cases autoGenerated is false, and certCAFilename is set (i.e. nonempty)

And (1) failed while (2) worked (I'm ignoring crashes inside TCMalloc now, it's probably unrelated to TLS configuration).

Is this right?

I looked at code, so far I see nothing suspicious. It seems some extra logging will be needed here.

TLS is mandatory

Yes, storage server always serves TLS, and agent always verifies TLS. I think we can relax this if there is a need.

soanni86 Feb 12, 2025
Author

i dont know what happened over night but a few agent pods (those that are standalone on dedicated cluster) started sending data and TLS error is gone (yes i used private CA in this case) ... other pods still fail due to tcmalloc :( i'm researching that issue

k -nperforator get po
NAME                     READY   STATUS             RESTARTS      AGE
perforator-agent-bvqsc   0/1     CrashLoopBackOff   3 (30s ago)   82s
perforator-agent-cntpg   0/1     CrashLoopBackOff   3 (41s ago)   82s
perforator-agent-d8rx7   0/1     Error              2 (26s ago)   48s
perforator-agent-hmd75   1/1     Running            0             102s
perforator-agent-j54kx   1/1     Running            0             54s
perforator-agent-jv5fj   0/1     CrashLoopBackOff   3 (25s ago)   80s
perforator-agent-w6m24   1/1     Running            0             49s
perforator-agent-wgbwv   0/1     CrashLoopBackOff   3 (38s ago)   80s

BigRedEye · 2025-02-12T09:33:59Z

BigRedEye
Feb 12, 2025
Maintainer

Regarding the scary stack trace from tcmalloc, it looks like google/tcmalloc#160. It's very strange that we haven't seen such errors in production. We will try to update tcmalloc soon to get rid of the errors. Can you provide more information about the frequency and stability of such errors to assess the impact? Do agents fail every time, or does the error occur with some probability?

2 replies

soanni86 Feb 12, 2025
Author

yeah its really interesting, the pods are failing due to

/home/runner/work/perforator/perforator/contrib/libs/tcmalloc/tcmalloc/internal/percpu_tcmalloc.h:1030] !hdr.IsLocked()

sporadically on different nodepools, looks not tied to any specific node

soanni86 Feb 12, 2025
Author

i added agents (configuration all the same) on my another cluster and all of them run just fine 🤔

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run perforator across dozens of clusters #34

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

run perforator across dozens of clusters #34

soanni86 Feb 10, 2025

Replies: 3 comments · 4 replies

MikailBag Feb 11, 2025 Maintainer

soanni86 Feb 11, 2025 Author

MikailBag Feb 12, 2025 Maintainer

soanni86 Feb 12, 2025 Author

BigRedEye Feb 12, 2025 Maintainer

soanni86 Feb 12, 2025 Author

soanni86 Feb 12, 2025 Author

soanni86
Feb 10, 2025

Replies: 3 comments 4 replies

MikailBag
Feb 11, 2025
Maintainer

soanni86
Feb 11, 2025
Author

MikailBag Feb 12, 2025
Maintainer

soanni86 Feb 12, 2025
Author

BigRedEye
Feb 12, 2025
Maintainer

soanni86 Feb 12, 2025
Author

soanni86 Feb 12, 2025
Author