Refactor input JSON support

[rgdeen]

🗒️ Summary

Support for JSON as an input file was partial at best. It allowed for access to top-level items but did not properly support structure such as arrays and maps. Fixed this by ditching most of the JSON-specific code (it tried to shoehorn the JSON into the PDS3Label structure to mimic PDS3 labels) and instead just registering the parse tree from the JSON parser directly with the Velocity engine. Turns out this Just Works, amazingly. One issue is that values were returned with quotes around them by the parsing engine. This was fixed by implementing a Velocity Uberspector to intercept get calls to the tree and dequote the values.

⚙️ Test Data and/or Report

Unfortunately, the "generate" wrapper script does not accept JSON input - this should be changed but is outside the scope. It is callable by the Labelocity wrapper (jConvertIIO) which is what I used for testing. Therefore I don't have a specific runnable test just for this package (and there is no general regression test). However, here's how to test using Labelocity.

$R2LIB/gen dummy.vic 10 10 ## really, any vicar file, it's not used but has to be present
java jpl.mipl.io.jConvertIIO inp=dummy.vic out=test_json.xml format=pds4 pds_detached_only=true pds_label_type=PDS4 velo_template=test_json.vm extra_file_name=test_json.json extra_file_type=json

See below for files.

♻️ Related Issues

TESTING FILES (GitHub would not allow me to attach .vm or .xml files)

====
test_json.json

{
  "Array": [ 1, 2, 3, 4, 5 ],
  "StrArray": [ "val1", "val2", "myval" ],
  "StructureArray": [
     {"a": 5, "b": 10, "c": "maybe"},
     {"a": 1, "b": 15, "c": "def"}
  ],

  "SCALPSS_Metadata": {
    "number_of_chunks": 18,
    "compression_type": "binned",
    "DSU_arrival_timestamp": "2024-02-18 07:28:44.362000",
    "non-volatile_storage_timestamp": "2024-02-18 07:28:44.374000",
    "capture_start_timestamp": "2024-02-18 07:28:39.553000",
    "altitude": 358999648,
    "frame_rate": 4,
    "capture_number": 1,
    "frame_number": 19,
    "pixel_data_corrupt": false,
    "camera_ID": 0,
    "pixel_data_checksum": 1171724560,
    "subarray": [ 100, 101, 102, 103 ],
    "subStrArray": [ "val1", "val2", "myval" ]
  },
  "Embedded_Camera_Metadata": {
    "timestamp": {
      "seconds": 12.618269088669951,
      "raw_register_value": 422912503,
      "register_fields": {
        "counts": 12,
        "cycles": 4946,
        "offset": 31
      }
    },
    "gain": {
      "decibels": 6.999164,
      "raw_register_value": 3271557190,
      "testing": "testval",
      "register_fields": {
        "presence": true,
        "control": "absolute",
        "one_push": "clear",
        "state": "on",
        "mode": "automatic",
        "value": 70
      }
    },
    "shutter": {
      "seconds": 0.010006,
      "raw_register_value": 3271557700,
      "register_fields": {
        "presence": true,
        "control": "absolute",
        "one_push": "clear",
        "state": "on",
        "mode": "automatic",
        "value": 580
      }
    },
    "brightness": {
      "percent": 5.859375,
      "raw_register_value": 3254780144,
      "register_fields": {
        "presence": true,
        "control": "absolute",
        "one_push": "clear",
        "state": "on",
        "mode": "manual",
        "value": 240
      }
    },
    "auto_exposure": {
      "EV": 0.624512,
      "raw_register_value": 3271557416,
      "register_fields": {
        "presence": true,
        "control": "absolute",
        "one_push": "clear",
        "state": "on",
        "mode": "automatic",
        "value": 296
      }
    },
    "white_balance": 0,
    "frame_counter": 25,
    "strobe_pattern": 2147483904,
    "GPIO_pin_state": 805306368,
    "ROI_position": 0
  },
  "MISSION_NAME": "scalpss"
}

====
test_json.vm

<?xml version="1.0" encoding="UTF-8"?>

  <Observation_Area>

<extra>$extra</extra>


<mission>$extra.MISSION_NAME</mission>
<fr>$extra.SCALPSS_Metadata</fr>
<fr>$extra.SCALPSS_Metadata.frame_number</fr>
<array>$extra.Array</array>
<array>$extra.Array[3]</array>
<strarray>$extra.StrArray[2]</strarray>
#set ($node = $extra.SCALPSS_Metadata)
<node>$node.compression_type</node>
<nodeclass>$node.getClass().getName().toString()</nodeclass>


#set ($node = $extra.Embedded_Camera_Metadata)
<node>$node</node>
<nodeclass>$node.getClass().getName().toString()</nodeclass>
<nodesub>$node.gain</nodesub>
<nodesubsub>$node.gain.decibels</nodesubsub>
<nodesubsub2>$node.gain.register_fields.control</nodesubsub2>
<nodesubsub2>$node.gain.register_fields.state</nodesubsub2>
<nodesubsub2>$node.gain.raw_register_value</nodesubsub2>
<nodesubsub2>$node.gain.testing</nodesubsub2>
#set ($db = $node.gain.decibels)
<db>$db.getClass()</db>
<db>$db</db>
<ns>$node.auto_exposure.EV</ns>
#set ($ctrl = $node.gain.register_fields.control)
<ctrl>$ctrl</ctrl>

<deep>$extra.Embedded_Camera_Metadata.brightness.percent</deep>
<struc>$extra.StructureArray[1]</struc>
<struc>$extra.StructureArray[1].b</struc>
<struc>$extra.StructureArray[1].c</struc>
<subarr>$extra.SCALPSS_Metadata.subarray</subarr>
<subarrv>$extra.SCALPSS_Metadata.subarray[2]</subarrv>
<subarrs>$extra.SCALPSS_Metadata.subStrArray</subarrs>
<subarrvs>$extra.SCALPSS_Metadata.subStrArray[2]</subarrvs>
#set ($list = $extra.getList("Embedded_Camera_Metadata"))
<foo>$list.toString()</foo>
#foreach ( $childNode in $list )
  <frclass> $childNode.getClass().getName().toString()</frclass>
  <frval> $childNode.toString()</frval>
#end

</Observation_Area>

====
test_json.xml (output)

<?xml version="1.0" encoding="UTF-8" standalone="no"?>


<Observation_Area>
  <extra>{"Array":[1,2,3,4,5],"StrArray":["val1","val2","myval"],"StructureArray":[{"a":5,"b":10,"c":"maybe"},{"a":1,"b":15,"c":"def"}],"SCALPSS_Metadata":{"number_of_chunks":18,"compression_type":"binned","DSU_arrival_timestamp":"2024-02-18 07:28:44.362000","non-volatile_storage_timestamp":"2024-02-18 07:28:44.374000","capture_start_timestamp":"2024-02-18 07:28:39.553000","altitude":358999648,"frame_rate":4,"capture_number":1,"frame_number":19,"pixel_data_corrupt":false,"camera_ID":0,"pixel_data_checksum":1171724560,"subarray":[100,101,102,103],"subStrArray":["val1","val2","myval"]},"Embedded_Camera_Metadata":{"timestamp":{"seconds":12.618269088669951,"raw_register_value":422912503,"register_fields":{"counts":12,"cycles":4946,"offset":31}},"gain":{"decibels":6.999164,"raw_register_value":3271557190,"testing":"testval","register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":70}},"shutter":{"seconds":0.010006,"raw_register_value":3271557700,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":580}},"brightness":{"percent":5.859375,"raw_register_value":3254780144,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"manual","value":240}},"auto_exposure":{"EV":0.624512,"raw_register_value":3271557416,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":296}},"white_balance":0,"frame_counter":25,"strobe_pattern":2147483904,"GPIO_pin_state":805306368,"ROI_position":0},"MISSION_NAME":"scalpss"}</extra>
  <mission>scalpss</mission>
  <fr>{"number_of_chunks":18,"compression_type":"binned","DSU_arrival_timestamp":"2024-02-18 07:28:44.362000","non-volatile_storage_timestamp":"2024-02-18 07:28:44.374000","capture_start_timestamp":"2024-02-18 07:28:39.553000","altitude":358999648,"frame_rate":4,"capture_number":1,"frame_number":19,"pixel_data_corrupt":false,"camera_ID":0,"pixel_data_checksum":1171724560,"subarray":[100,101,102,103],"subStrArray":["val1","val2","myval"]}</fr>
  <fr>19</fr>
  <array>[1,2,3,4,5]</array>
  <array>4</array>
  <strarray>myval</strarray>
  <node>binned</node>
  <nodeclass>com.fasterxml.jackson.databind.node.ObjectNode</nodeclass>
  <node>{"timestamp":{"seconds":12.618269088669951,"raw_register_value":422912503,"register_fields":{"counts":12,"cycles":4946,"offset":31}},"gain":{"decibels":6.999164,"raw_register_value":3271557190,"testing":"testval","register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":70}},"shutter":{"seconds":0.010006,"raw_register_value":3271557700,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":580}},"brightness":{"percent":5.859375,"raw_register_value":3254780144,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"manual","value":240}},"auto_exposure":{"EV":0.624512,"raw_register_value":3271557416,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":296}},"white_balance":0,"frame_counter":25,"strobe_pattern":2147483904,"GPIO_pin_state":805306368,"ROI_position":0}</node>
  <nodeclass>com.fasterxml.jackson.databind.node.ObjectNode</nodeclass>
  <nodesub>{"decibels":6.999164,"raw_register_value":3271557190,"testing":"testval","register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":70}}</nodesub>
  <nodesubsub>6.999164</nodesubsub>
  <nodesubsub2>absolute</nodesubsub2>
  <nodesubsub2>on</nodesubsub2>
  <nodesubsub2>3271557190</nodesubsub2>
  <nodesubsub2>testval</nodesubsub2>
  <db>class com.fasterxml.jackson.databind.node.DoubleNode</db>
  <db>6.999164</db>
  <ns>0.624512</ns>
  <ctrl>absolute</ctrl>
  <deep>5.859375</deep>
  <struc>{"a":1,"b":15,"c":"def"}</struc>
  <struc>15</struc>
  <struc>def</struc>
  <subarr>[100,101,102,103]</subarr>
  <subarrv>102</subarrv>
  <subarrs>["val1","val2","myval"]</subarrs>
  <subarrvs>myval</subarrvs>
  <foo>[{"seconds":12.618269088669951,"raw_register_value":422912503,"register_fields":{"counts":12,"cycles":4946,"offset":31}}, {"decibels":6.999164,"raw_register_value":3271557190,"testing":"testval","register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":70}}, {"seconds":0.010006,"raw_register_value":3271557700,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":580}}, {"percent":5.859375,"raw_register_value":3254780144,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"manual","value":240}}, {"EV":0.624512,"raw_register_value":3271557416,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":296}}, 0, 25, 2147483904, 805306368, 0]</foo>
  <frclass> com.fasterxml.jackson.databind.node.ObjectNode</frclass>
  <frval> {"seconds":12.618269088669951,"raw_register_value":422912503,"register_fields":{"counts":12,"cycles":4946,"offset":31}}</frval>
  <frclass> com.fasterxml.jackson.databind.node.ObjectNode</frclass>
  <frval> {"decibels":6.999164,"raw_register_value":3271557190,"testing":"testval","register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":70}}</frval>
  <frclass> com.fasterxml.jackson.databind.node.ObjectNode</frclass>
  <frval> {"seconds":0.010006,"raw_register_value":3271557700,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":580}}</frval>
  <frclass> com.fasterxml.jackson.databind.node.ObjectNode</frclass>
  <frval> {"percent":5.859375,"raw_register_value":3254780144,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"manual","value":240}}</frval>
  <frclass> com.fasterxml.jackson.databind.node.ObjectNode</frclass>
  <frval> {"EV":0.624512,"raw_register_value":3271557416,"register_fields":{"presence":true,"control":"absolute","one_push":"clear","state":"on","mode":"automatic","value":296}}</frval>
  <frclass> com.fasterxml.jackson.databind.node.IntNode</frclass>
  <frval> 0</frval>
  <frclass> com.fasterxml.jackson.databind.node.IntNode</frclass>
  <frval> 25</frval>
  <frclass> com.fasterxml.jackson.databind.node.LongNode</frclass>
  <frval> 2147483904</frval>
  <frclass> com.fasterxml.jackson.databind.node.IntNode</frclass>
  <frval> 805306368</frval>
  <frclass> com.fasterxml.jackson.databind.node.IntNode</frclass>
  <frval> 0</frval>
</Observation_Area>

Resolves #84

[jordanpadams]

@rgdeen in the future, when trying to attach files that GitHub doesn't like, we tend to just group them together and create a zip

[rgdeen]

@jordanpadams here ya go...

milabel_json_test.tar.gz

[jordanpadams]

The comments are not really using the appropriate coding standard to be picked up by apidocs. Also, no tests provided so really no way to ensure this code is successful, but considering I am fairly certain this code is strictly used by labelocity, I am going to merge this once we get some test data.

@rgdeen can you please upload a set of example templates/data files and how to manually run in order to test that this works? src/test/resources/ works to put your test data.

[jordanpadams]

@rgdeen actually, I see that you copy-pasted those files into the PR comments, but can we actually upload these to the repo?

[jordanpadams]

and add them here: https://github.com/NASA-PDS/mi-label/tree/main/src/test/resources

[sonarqubecloud]

Quality Gate passed

Issues
15 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.7% Duplication on New Code

See analysis details on SonarQube Cloud