Support for StringIO

[awicenec]

JIRA Ticket

liu-418

Type

• Feature (addition)
• Bug fix
• Refactor (change)
• Documentation

Problem/Issue

Provide ability to read and write plain strings if requested by the user.

Solution

The basic solution is to use StringIO rather than BytesIO if requested by the user, but the IO implementation of DALiuGE is quite complex and requires a number of changes to make this happen. As a side-effect the functionality to provide data for a memory drop through the pydata field is now sort of type-safe and obeying the type specified in the pull-down menu. Moreover, when copying from one drop to another the original type of the data is preserved. As a side effect, when copying from one MemoryDrop to another the types of those drops need to match. This is a bit of a hidden feature right now and can lead to unexpected graph execution failures, since the way to specify the type of the target MemoryDrop is by using the pydata type pull-down menu, but leaving the pydata value empty. We might want to think abut a more explicit way of specifying the type of a MemoryDrop in the future.

Checklist

No specific documentation has been added.

• Unittests added
  • The existing unit tests are covering a good part of this functionality already. We should probably add a specific graph for the more edge cases.
• Documentation added
  • In particular the 'type checking' side effect should be documented, but since we should probably add an explicit type to DataDROPs more generally that will be the right time to document the whole story.

Summary by Sourcery

Add support for StringIO to enable reading and writing of plain strings, enhancing type safety and data type preservation during data operations.

New Features:

• Introduce support for reading and writing plain strings using StringIO instead of BytesIO when specified by the user.

Enhancements:

• Ensure type safety and preservation of data types when copying data between drops, with a focus on maintaining the original data type.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR adds support for StringIO to handle string data alongside BytesIO in the DALiuGE engine. The implementation includes type-safe handling of pydata fields, preservation of data types when copying between drops, and UTF-8 string handling capabilities. The changes span across multiple components including data drops, IO handling, and data parsing.

Sequence diagram for data reading and writing with StringIO

sequenceDiagram
    participant User
    participant DataDROP
    participant DataIO
    participant StringIO

    User->>DataDROP: Request to read data
    DataDROP->>DataIO: getIO()
    DataIO->>StringIO: open(OpenMode.OPEN_READ)
    StringIO-->>DataIO: Return data
    DataIO-->>DataDROP: Return data
    DataDROP-->>User: Return data

    User->>DataDROP: Request to write data
    DataDROP->>DataIO: getIO()
    DataIO->>StringIO: open(OpenMode.OPEN_WRITE)
    StringIO-->>DataIO: Write data
    DataIO-->>DataDROP: Confirm write
    DataDROP-->>User: Confirm write

Loading

Class diagram for updated IO handling

classDiagram
    class DataIO {
        +open(mode: OpenMode)
        +read(count: int)
        +write(data)
        +close()
    }
    class MemoryIO {
        +_open()
        +_write(data)
        +exists()
        +delete()
    }
    class FileIO {
        +_read(count: int)
        +_write(data)
    }
    DataIO <|-- MemoryIO
    DataIO <|-- FileIO

    class DropParser {
        <<enumeration>>
        +PATH
        +DATAURL
        +UTF8
    }

    class DropLoader {
        +load_utf8(drop: DataDROP)
    }

Loading

File-Level Changes

Change	Details	Files
Added StringIO support in memory drops with type safety	Added string type handling in parse_pydata function Modified initialize method to use StringIO based on pydata type Added type checking when copying between MemoryDrops Added handling for empty pydata values	daliuge-engine/dlg/data/drops/memory.py
Enhanced IO operations with string handling capabilities	Added string to bytes conversion for checksum calculation Added StringIO support in _open method Added UTF-8 encoding for string data in file operations	daliuge-engine/dlg/data/drops/data_base.py daliuge-engine/dlg/data/io.py
Added UTF-8 parsing and loading functionality	Added UTF8 enum to DropParser Implemented load_utf8 function for string data loading Added UTF-8 handling in write_results method	daliuge-engine/dlg/named_port_utils.py daliuge-engine/dlg/drop_loaders.py daliuge-engine/dlg/apps/pyfunc.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @awicenec - I've reviewed your changes - here's some feedback:

Overall Comments:

• Please add unit tests covering the new StringIO functionality and type checking behavior between MemoryDrops to prevent potential regressions.
• The new type checking behavior between MemoryDrops needs to be documented to prevent unexpected runtime failures. Please add documentation explaining this functionality to users.

Here's what I looked at during the review

• 🟡 General issues: 2 issues found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (code-quality): Raise a specific error instead of the general Exception or BaseException (raise-specific-error)

Explanation

If a piece of code raises a specific exception type rather than the generic [`BaseException`](https://docs.python.org/3/library/exceptions.html#BaseException) or [`Exception`](https://docs.python.org/3/library/exceptions.html#Exception), the calling code can:

• get more information about what type of error it is
• define specific exception handling for it

This way, callers of the code can handle the error appropriately.

How can you solve this?

• Use one of the built-in exceptions of the standard library.
• Define your own error class that subclasses Exception.

So instead of having code raising Exception or BaseException like

if incorrect_input(value):
    raise Exception("The input is incorrect")

you can have code raising a specific error like

if incorrect_input(value):
    raise ValueError("The input is incorrect")

or

class IncorrectInputError(Exception):
    pass


if incorrect_input(value):
    raise IncorrectInputError("The input is incorrect")

[coveralls]

coverage: 79.724% (-0.03%) from 79.752%
when pulling 22a2740 on liu-418
into c09b85e on master.

[myxie]

Hi @awicenec, I've added some suggestions for the Exceptions. Everything looks good.

[myxie]

@awicenec the code is fine as it is if you'd like to get these changes merged. I think we can update the exception handling at a later date.