Some requests take >10s to complete (and many more >1s)

[ConnorNelson]

Methodology

Sampling from the last million requests, this finds requests that spent more than 10s in the first upstream (CTFd container):

docker logs -n1000000 nginx-proxy | grep -o '{.*}$' | jq -r '
  select(.upstream_response_time != "") |
  select(.upstream_response_time | split(" : ")[0] | tonumber > 10.0) |
  "\(.request)"
' | sort | uniq -c | sort -nr

We redact some of the results with REDACTED.

>10s

   1984 POST /api/v1/challenges/attempt HTTP/2.0
    189 GET /dojo/cse365-f2024/course/grades HTTP/2.0
    147 GET /dojo/cse365-f2024/course/syllabus HTTP/2.0
    111 GET /dojo/cse365-f2024/course HTTP/2.0
    107 GET /pwncollege_api/v1/workspace?service=code HTTP/2.0
     86 GET /cse365-f2024/ HTTP/2.0
     51 POST /pwncollege_api/v1/docker HTTP/2.0
     48 GET /intro-to-cybersecurity/ HTTP/2.0
     29 GET /dojo/cse365-f2024/course/setup HTTP/2.0
     19 GET /fundamentals/ HTTP/2.0
     18 GET /dojo/REDACTED/admin/canvas/sync HTTP/1.1
     18 GET /dojo/REDACTED/admin/canvas/sync HTTP/1.1
      9 POST /api/v1/challenges/attempt HTTP/1.1
      4 GET /dojo/cse365-f2024/course/setup? HTTP/2.0
      3 GET /cse365-f2024/ HTTP/1.1
      2 GET /pwncollege_api/v1/workspace?service=desktop HTTP/2.0
      2 GET /intro-to-cybersecurity/ HTTP/1.1
      2 GET /dojo/cse365-f2024/course/grades? HTTP/2.0
      2 GET /dojo/cse365-f2024/admin/grades HTTP/2.0
      1 GET /software-exploitation/kernel-exploitation HTTP/1.1
      1 GET /fundamentals/?ref=REDACTED HTTP/2.0
      1 GET /fundamentals/ HTTP/1.1
      1 GET /dojo/cse365-s2024/course/setup HTTP/2.0
      1 GET /dojo/cse365-f2024/course/syllabus HTTP/1.1
      1 GET /dojo/cse365-f2024/course/identity HTTP/2.0
      1 GET /dojo/cse365-f2024/admin/grades.csv HTTP/2.0
      1 GET /dojo/cse365-f2023/course HTTP/2.0
      1 GET /cse466-f2023/?ref=REDACTED HTTP/2.0
      1 GET /cse466-f2023 HTTP/2.0
      1 GET /cse466-f2022/ HTTP/1.1

>1s

In this case, we just look at requests that were made 100 or more times in that last million requests sample.

   8877 POST /pwncollege_api/v1/docker HTTP/2.0
   5649 GET /pwncollege_api/v1/workspace?service=code HTTP/2.0
   3890 POST /api/v1/challenges/attempt HTTP/2.0
    906 GET /pwncollege_api/v1/scoreboard/cse365-f2024/_/30/1 HTTP/2.0
    559 GET /pwncollege_api/v1/scoreboard/linux-luminarium/_/30/1 HTTP/2.0
    510 GET /pwncollege_api/v1/scoreboard/intro-to-cybersecurity/_/30/1 HTTP/2.0
    213 GET /pwncollege_api/v1/scoreboard/fundamentals/_/30/1 HTTP/2.0
    204 GET /pwncollege_api/v1/scoreboard/program-security/_/30/1 HTTP/2.0
    197 GET /dojo/cse365-f2024/course/grades HTTP/2.0
    180 GET /dojo/cse365-f2024/course/syllabus HTTP/2.0
    163 GET /hacker/ HTTP/2.0
    163 GET /cse365-f2024/ HTTP/2.0
    153 GET /pwncollege_api/v1/workspace?service=desktop HTTP/2.0
    151 GET /dojo/cse365-f2024/course HTTP/2.0
    149 GET /cse365-f2024/cryptography/ HTTP/2.0
    118 GET /fundamentals/program-misuse/ HTTP/2.0
    107 GET /linux-luminarium/ HTTP/2.0
    100 GET /intro-to-cybersecurity/talking-web/ HTTP/2.0

There are also several GET /hacker/N HTTP/2.0 requests (for various Ns), and many more sporadic requests (with <100 count) that we can investigate once we've addressed this current information.

Additionally, with an honorable mention, we had 44 GET / HTTP/2.0 that took >1s. We can't have a slow / ever!