Adapt Miyao Staff Finding for Aquitanian notation

[martha-thomae]

Adapt Miyao Staff Finding to work with just one staff line: the music reference line for Aquitanian music sources (see the red line between the neumes for each music system).

[DeannaLC]

Here's some documentation of running Miyao Staff Finding with the Number of Lines in settings from 0-8 (min value to max value, default value is 4)

The image used was staff lines from Einsiedeln Codex 366 009v classified with Pixel

The following Rodan workflow was used

Number of Lines = 0, 2, 3:

  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 77, in run_my_task
    staves = sf.get_staves(image)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 37, in get_staves
    self._find_staves(s)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 366, in _find_staves
    diff_lo = avg_lines[3] - avg_lines[2]
IndexError: list index out of range

Number of Lines = 1:

  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 77, in run_my_task
    staves = sf.get_staves(image)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 37, in get_staves
    self._find_staves(s)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 312, in _find_staves
    top_second_start_y = lines_top[1][0][1]
IndexError: list index out of range

Number of Lines = 4:
Successfully generates the following JSON output:
Miyao Staff Finding - JSOMR.json

'Number of Lines = 5, 6, 7, 8':

  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 77, in run_my_task
    staves = sf.get_staves(image)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 37, in get_staves
    self._find_staves(s)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 267, in _find_staves
    raise Exception("Gamera cannot find staves with threshold blackness 0.3")
Exception: Gamera cannot find staves with threshold blackness 0.3

[DeannaLC]

Here's the results of testing Number of Staff Lines from 0 to 8 using the reference line/scrape classified from Folio 029v Seq 001 in the Salamanca Missal

Image used:

Number of Lines = 0
Miyao Staff Finding - JSOMR.json

Number of Lines = 1, 2

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 77, in run_my_task
    staves = sf.get_staves(image)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 37, in get_staves
    self._find_staves(s)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 267, in _find_staves
    raise Exception("Gamera cannot find staves with threshold blackness 0.3")
Exception: Gamera cannot find staves with threshold blackness 0.3

Number of Lines = 3

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/base.py", line 77, in run_my_task
    staves = sf.get_staves(image)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 37, in get_staves
    self._find_staves(s)
  File "/code/Rodan/rodan/jobs/heuristic_pitch_finding/StaffFinding.py", line 366, in _find_staves
    diff_lo = avg_lines[3] - avg_lines[2]
IndexError: list index out of range

Number of Lines = 4
Miyao Staff Finding - JSOMR.json

Number of Lines = 5
Miyao Staff Finding - JSOMR.json

Number of Lines = 6
Miyao Staff Finding - JSOMR.json

Number of Lines = 7
Miyao Staff Finding - JSOMR.json

Number of Lines = 8
Miyao Staff Finding - JSOMR.json

[fujinaga]

You should be using "Number of Lines = 1" at all times, because it refers to the number of staff lines per stave. In Aquitanian, you only have one staff line per stave. It may be quicker to solve this problem, if you use an independent (i.e., non-Gamera) algorithm to find line segments on a page. Use miyao.py but do a special case when "Number of Lines == 1" Ask ChatGPT for help finding algorithms then use Code Pilot to write and debug your code. You need to worry about the case when a staff lines may be broken up into shorter line segments. This may happen when we try to detect the staff lines in red automatically later.

[DeannaLC]

You should be using "Number of Lines = 1" at all times, because it refers to the number of staff lines per stave. In Aquitanian, you only have one staff line per stave.

Got it. I wanted to just test first to see how Miyao Staff Finding would work for different numbers of staff lines, but will keep it at 1 from now on.

It may be quicker to solve this problem, if you use an independent (i.e., non-Gamera) algorithm to find line segments on a page. Use miyao.py but do a special case when "Number of Lines == 1"

I could do a similar method to when I was working on finding the reference line, which used cv2 HoughLines or bounding boxes. I'm still figuring out exactly what the JSOMR is outputting so I know what information I need to put into it.

This is the output I can get by running cv2.HoughLines after some filtering. I used lines classified from 029v Seq 001. Let me know what you think or if there's a different approach you were thinking of.

@fujinaga @martha-thomae

[martha-thomae]

Thanks for the feedback @fujinaga!

@DeannaLC, for this is very important to follow the actual path of the staff line. So, if the page curves and, because of that, the music reference line curves, we need to be able to follow this curvature as best as possible; otherwise, the prediction of the pitches will be all wrong. So, rather than using a straight line (like in this example of cv2.HoughLines), you need to basically take a few samples (points) on the reference line, and then interpolate them. Basically drawing smaller line segments to join the points (samples) of the staff line. This is what Miyao Staff Finding is supposed to do. So either you can try to adapt it, or just program something that does this exact same thing for one staff line.

[DeannaLC]

Got it, I think for the time being I'm leaning towards using line detection algorithms that aren't Miyao since I was struggling a bit to understand what's already implemented.

One approach I'm experimenting with is using cv2.minAreaRect which draws a bounding box around the staff line, then drawing a line that goes through it. With the few cases I've tried so far it seems much more accurate than HoughLines. I can experiment with it more and try to split each staff into 4-8 sections.

These are some results after preprocessing with one-bit, despeckle, and dilate.

015v Seq 001

029v Seq 001

055v Seq 001

[DeannaLC]

The current solution I've come up with uses bounding boxes around each reference line, then divides it into a certain number of slices (default I'm using is 8). Then I draw a box around each segment and connect the segments together.

Sample result using 015v Seq 001:

Results as a JSON file: Lines.json

For now I'm going to make a separate branch and have it as its own job.

@fujinaga

Use miyao.py but do a special case when "Number of Lines == 1"

Did you mean for this to be a part of Miyao Staff Finding (job with JSON output that's part of full workflow) or Miyao Staff Finder (job with PNG output)? As well, are we hoping for it to just be a separate case of the existing job as opposed to a new job?

[martha-thomae]

@DeannaLC, could you please clarify which is the original staff line in the manuscript (which you draw with Pixel given that it was a nearly invisible line) and which is the line predicted based on the 8 divisions you made?

I guess what is in black is the line you draw in pixel (the ground truth), and the red lines are the prediction of the line after dividing it into 8 segments and try to predict the individual parts, right?

And summarising your process here:

• Split the staff line into 8 sections
• Draw a bounding box around each of those sections
• Draw a line that goes through the through the center of each bounding box (using cv2.minAreaRect)
• The 8 lines are then connected together

If there are any inaccuracies here, please let us know :)
Thank you, @DeannaLC!

[DeannaLC]

Sorry yes, the ground truth line that I classified from Pixel is the black line, and the red line is the one that's being drawn in.

I would describe the steps as the following:

1. Use cv2.boundingRect to draw straight/all 90 degree angle rectangular bounding boxes around each reference line
2. Split each of the bounding rectangles into 8 sections
3. Use cv2.minAreaRect (slanted rectangles) around each of the smaller sections, then draw a line through the center of each
4. Connect the lines together

I've also added a branch called one-staff-finding which contains a One Staff Finding job. It takes in a png of the classified reference line and returns a JSOMR and optionally a png that has the image with the lines drawn over it.