Where do you position yourself compared to Pyroscope?

[PavelPikat]

Continuous profiling isn't new, e.g. there is already a wildly popular and production ready Grafana Pyroscope that can profile not just CPU, but memory, allocations, locks etc

How do you compare yourself to Pyroscope and why should the community use Perforator? What new ideas does it bring? What problems does it solve that Pyroscope doesnt?

[BigRedEye]

Great question! Perforator indeed looks similar to Pyroscope. However, we think that the closest existing solutions are https://parca.dev, closed-source Google Wide Profiling, and, speaking of the agent, the beautiful OpenTelemetry eBPF profiler. The main technical differences with Pyroscope we see are:

• Pyroscope's Java support is superior as of now because Pyroscope offloads it to the amazing async-profiler.
• Pyroscope expects native binaries to be compiled with frame pointers: https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/ebpf/troubleshooting/#understand-flat-stack-traces. This is often not the case, and that's the problem we've tried to solve with Perforator. Perforator uses .eh_frame, which is nearly universal and does not impose additional requirements on compiled binaries.
• Pyroscope symbolizes using symtab: https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/ebpf/troubleshooting/#troubleshoot-unknown-symbols. We use DWARF/GSYM to get as correct and verbose stacks as possible (we benchmark our stacks against stacks from gdb).
• Pyroscope symbolizes profiles on an agent, while Perforator symbolizes profiles offline, greatly reducing symbolization costs and agent's overhead. It seems Pyroscope is heading toward the same architecture we use: Prototype(symbolization): Add symbolization in Pyroscope read path grafana/pyroscope#3799.
• Perforator can be (and should be!) run as a standalone replacement for perf record.
• Perforator supports sPGO profiles.

In summary, we try to implement native profiling almost perfectly. It's worth noting that Pyroscope is a mature, well-established product that integrates excellently with the Grafana ecosystem. We have just focused on different things: our focus has been on optimizing native code profiling and making it as accurate and low-overhead as possible.