Updating to UBI9.5 and Ruby 3.3.7

[stephen-ritter]

Due to security reasons I was attempting to update to UBI9.5, however I hit some issues. I hit an OpenSSL issue when I ran the line gem update --system 3.3.14:

OpenSSL is not available. Install OpenSSL and rebuild Ruby or use non-HTTPS sources (Gem::Exception)

To fix this I updated Ruby to 3.3.7, however I was faced with another issue in that the unit tests started to fail. Specifically the tests in ruby_lex_utils_spec.rb were failing as it appears that the class RubyLex has been refactored heavily:
https://msp-greg.github.io/ruby_3_3/irb/IRB/RubyLex.html
https://msp-greg.github.io/ruby_3_2/irb/RubyLex.html
I did some work to fix some of the tests by switching from using RubyLex to Prism. I have been stuck on the function each_lexed_segment as Prism for the most part has been a drop in replacement however in the implementation of this function it is a bit trickier. I am attaching the code that I have updated just to give it as an example, and it may be helpful in a future update.

It should be noted that Prism does not emit the on_sp event, so you will not be able to recreate your code with the spaces in it, thus some of the tests will need to be updated to account for this lack of spacing.
ruby/prism#722

# encoding: ascii-8bit

# Copyright 2022 Ball Aerospace & Technologies Corp.
# All Rights Reserved.
#
# This program is free software; you can modify and/or redistribute it
# under the terms of the GNU Affero General Public License
# as published by the Free Software Foundation; version 3 with
# attribution addendums as found in the LICENSE.txt
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.

# Modified by OpenC3, Inc.
# All changes Copyright 2022, OpenC3, Inc.
# All Rights Reserved
#
# This file may also be used under the terms of a commercial license
# if purchased from OpenC3, Inc.

require 'prism'

class RubyLexUtils
  # Regular expression to detect blank lines
  BLANK_LINE_REGEX  = /^\s*$/
  # Regular expression to detect lines containing only 'else'
  LONELY_ELSE_REGEX = /^\s*else\s*$/

  KEY_KEYWORDS = [
    'class'.freeze,
    'module'.freeze,
    'def'.freeze,
    'undef'.freeze,
    'begin'.freeze,
    'rescue'.freeze,
    'ensure'.freeze,
    'end'.freeze,
    'if'.freeze,
    'unless'.freeze,
    'then'.freeze,
    'elsif'.freeze,
    'else'.freeze,
    'case'.freeze,
    'when'.freeze,
    'while'.freeze,
    'until'.freeze,
    'for'.freeze,
    'break'.freeze,
    'next'.freeze,
    'redo'.freeze,
    'retry'.freeze,
    'in'.freeze,
    'do'.freeze,
    'return'.freeze,
    'alias'.freeze
  ]

  # @param text [String]
  # @return [Boolean] Whether the text contains the 'begin' keyword
  def contains_begin?(text)
    lex = Prism.lex(text)
    tokens = lex.value
    tokens.each do |token|
      token_object = token[0]
      if token_object.type == :KEYWORD_BEGIN
        return true
      end
    end
    return false
  end

  # @param text [String]
  # @return [Boolean] Whether the text contains the 'end' keyword
  def contains_end?(text)
    lex = Prism.lex(text)
    tokens = lex.value
    tokens.each do |token|
      token_object = token[0]
      if token_object.type == :KEYWORD_END
        return true
      end
    end
    return false
  end

  # @param text [String]
  # @return [Boolean] Whether the text contains a Ruby keyword
  def contains_keyword?(text)
    lex = Prism.lex(text)
    tokens = lex.value
    tokens.each do |token|
      token_object = token[0]
      state_bits = token[1]
      if token_object.type.start_with?("KEYWORD")
        if KEY_KEYWORDS.include?(token_object.value)
          return true
        end
      elsif token_object.type == :BRACE_LEFT and state_bits != (Ripper::EXPR_BEG | Ripper::EXPR_LABEL)
        return true
      end
    end
    return false
  end

  # @param text [String]
  # @return [Boolean] Whether the text contains a keyword which starts a block.
  #   i.e. 'do', '{', or 'begin'
  def contains_block_beginning?(text)
    lex = Prism.lex(text)
    tokens = lex.value
    tokens.each do |token|
      token_object = token[0]
      if token_object.type == :KEYWORD_BEGIN || token_object.type == :KEYWORD_DO || token_object.type == :BRACE_LEFT
        return true
      end
    end
    return false
  end

  def continue_block?(text)
    lex = Prism.lex(text)
    tokens = lex.value
    index = tokens.length - 1
    while index > 0
      token_object = tokens[index][0]
      return true if token_object.type == :KEYWORD_DO
      index -= 1
    end
    return false
  end

  # @param text [String]
  # @param progress_dialog [OpenC3::ProgressDialog] If this is set, the overall
  #   progress will be set as the processing progresses
  # @return [String] The text with all comments removed
  def remove_comments(text, progress_dialog = nil)
    lex = Prism.lex(text)
    tokens = lex.value
    comments_removed = ""
    token_count = 0
    progress = 0.0
    tokens.each do |token|
      token_object = token[0]
      token_count += 1
      if token_object.type != :COMMENT
        comments_removed << token_object.value
      else
        newline_count = token_object.value.count("\n")
        comments_removed << ("\n" * newline_count)
      end
      if progress_dialog and token_count % 10000 == 0
        progress += 0.01
        progress = 0.0 if progress >= 0.99
        progress_dialog.set_overall_progress(progress)
      end
    end
    return comments_removed
  end

[jmthomas]

Thanks @stephen-ritter! We've been specifically holding off on upgrading Ruby due to the changes with IRB. Prism is the right answer and we recently updated the python parsing in script_instrumentor.py which should be very similar to how the new Prism code works.

We currently have an open ticket #1579 to address the Ruby upgrade. This isn't a high priority for us but should be implemented by the end of Q2 2025.