Test how fast PGEN can consume the table

[tomtrafford]

First step of development for #61.

We would like to understand and test the current functionality of the PGEN table. We want to write a large PGEN table and document the metrics of how fast the table is consumed.

[tomtrafford]

After having discussed this with @coretl
We want to test the capabilities of writing a table through the DMA interface. To do this we would want to measure how long it takes to write a very large table (10 million points) to the server. The best interface for this would be using the PandABlocks python client, testing in both base64 and ascii formats.

We want to check how this affects the CPU usage of the PandA server during this process, and ensure that if we write points to PGEN at 1MHz that PCAP can capture them at 1MHz.

[adedamola-sode]

@tomtrafford @coretl Here's the timing data for the panda sending instructions. Blue, simple put command to set a bit to 1, Red, Generating and sending lists of data to PGEN. You can also scrutinise the code here: https://github.com/adedamola-sode/panda_test_scripts_issue62.

[coretl]

Please can you put the timer and the for loop around just the inner function (the client.send), so we aren't making and tearing down a connection for each run

Also, the PGEN script is doing multiple Puts to PGEN within a gather, this means they will happen in parallel, while we want to measure them in series. Please can you remove the gather and await each client.send in turn, checking how long that takes

Finally, I'm not sure what the graph is showing, why does it drop to almost zero every other point?

[adedamola-sode]

I'll make the changes and test further. It's 10 points of data for 10 iterations of the main loop, so for some reason, it's drastically little time every other function call? I'm not sure why this behaviour is there. It was weird, so I thought a graph would be good to show it being weird

[adedamola-sode]

I'll send over the data on slack, or make a page for you to view it. basic averages are 0.0003 seconds for a single put, and 0.8 seconds for the pgen sending 1000000, points of data

[coretl]

Excellent, that's the results I wanted to see, so with a few optimisations we can probably stream points at 1MHz to PGEN, which was the target.

Have you checked that you can produce those points and see them captured in PCAP?