Record lines truncation of target PDS Enhancement #2433

pujal0909 · 2025-02-09T17:36:17Z

What It Does

In the scenario when copying from a source PDS to a target PDS where the target does not have enough record lines for a member, initially, it would copy over that member, truncate it's record length, throw a truncation error and would not continue copying over the subsequent members. Now, it will continue copying over the subsequent members and inform the user that member(s)' contents were truncated and they can view the list of members affected in a local file.

How to Test

With the current functionality, copy a source PDS into a target PDS. The target PDS should have a lower record lines value than the source.
Notice that a member with a long record length will be truncated, and subsequent members will not continue to copy over.
Now with the updated functionality, copy the same source PDS into a target PDS with a lower record lines value than the source.
Notice that the user is informed of the truncation and is given a local file of members with this error. The subsequent members were copied over as well.

Review Checklist
I certify that I have:

tested my changes
added/updated automated tests
updated the changelog
followed the contribution guidelines

Additional Comments

Should this technically be a bug fix?
Should the Github security suggestion about using the tmp library be ignored?

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

codecov · 2025-02-09T17:48:52Z

Codecov Report

Attention: Patch coverage is 90.69767% with 4 lines in your changes missing coverage. Please review.

Project coverage is 91.50%. Comparing base (b91b989) to head (e35a30d).
Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
packages/zosfiles/src/methods/copy/Copy.ts	90.24%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2433      +/-   ##
==========================================
- Coverage   91.50%   91.50%   -0.01%     
==========================================
  Files         641      641              
  Lines       18376    18414      +38     
  Branches     3872     3885      +13     
==========================================
+ Hits        16815    16849      +34     
- Misses       1559     1562       +3     
- Partials        2        3       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts

packages/zosfiles/src/methods/copy/Copy.ts

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

Signed-off-by: Pujal Gandhi <71276682+pujal0909@users.noreply.github.com>

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/zosfiles/src/methods/copy/Copy.ts

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-actions · 2025-02-13T13:24:34Z

📅 Suggested merge-by date: 2/27/2025

packages/zosfiles/src/methods/copy/Copy.ts

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/cli/CHANGELOG.md

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

Misread conditional logic

traeok

Changes LGTM, thanks @pujal0909 for adding the truncation warning!

zFernand0

LGTM! 😋

I'm curious to know if it would be possible to add a progress bar.
Trying to copy a PDS with almost 200 members looks like the command hands 😋

Also... I'm afraid that some errors are not being shown 🤔

As a final test, I let it run and here is what happened:

Here you can see that the command result only mentions the members that it was able to copy. (24 out of 193)
But it didn't mention the members that were not copied. likely due to not having enough space on the target PDF 🙏

All of that said, I do believe that this PR does handle quite a bit of scenarios and we could tackle these edge cases in a separate issue 🙏

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/cli/src/zosfiles/copy/ds/Ds.handler.ts

zFernand0 · 2025-02-26T16:16:18Z

packages/zosfiles/src/methods/copy/Copy.ts

+            const truncatedMembersFile = path.join(tmpdir(), 'truncatedMembers.txt');
+            if(truncatedMembers.length > 0) {
+                const firstTenMembers = truncatedMembers.slice(0, 10);
+                fs.writeFileSync(truncatedMembersFile, truncatedMembers.join('\n'), {flag: 'w'});


We've already determined that we are fine with writing the list of truncated members in a temporary file : )

Ideally we would make sure the file permissions are 600

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/cli/src/zosfiles/copy/ds/Ds.handler.ts

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-actions · 2025-02-25T11:03:07Z

Reminder: This pull request has a merge-by date coming up within the next 24 hours. Please review this PR as soon as possible.

@awharn @adam-wolfe @jace-roell @ATorrise @t1m0thyj @anaxceron

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

anaxceron

Left a couple comments re: changelog.

packages/cli/CHANGELOG.md

packages/zosfiles/CHANGELOG.md

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts

anaxceron

Nice changelogs! Thanks @pujal0909!

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

sonarqubecloud · 2025-02-25T17:36:55Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

zFernand0 · 2025-02-26T16:15:10Z

Found this issue while testing, but this also on the master branch

> zdev files copy ds DS.DOES.NOT.EXIST fernando.test.pds.small      [±truncate-lrecl-target ✓(✹)]
Unable to perform this operation due to the following problem.
Cannot read properties of undefined (reading 'dsorg')

Response From Service
TypeError: Cannot read properties of undefined (reading 'dsorg')

I believe we will need to handle missing datasets in the isPDS function, and maybe fail a bit more gracefully 🙏

I believe these requested-changes were addressed 🙏

truncate lrecl functionality

2016201

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-actions bot assigned pujal0909 Feb 9, 2025

pujal0909 changed the title ~~Truncate lrecl target~~ Record lines truncation error - target data set Feb 9, 2025

updated system tests

8d2a989

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

pujal0909 changed the title ~~Record lines truncation error - target data set~~ Record lines truncation of target data set Enhancement Feb 10, 2025

pujal0909 changed the title ~~Record lines truncation of target data set Enhancement~~ Record lines truncation of target PDS Enhancement Feb 11, 2025

pujal0909 added 2 commits February 11, 2025 20:22

updated changelogs

37a2941

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

updated system tests

dd84ed7

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-advanced-security bot found potential problems Feb 12, 2025

View reviewed changes

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show fixed Hide fixed

updated temp file removal

c62d262

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

pujal0909 force-pushed the truncate-lrecl-target branch from a439165 to c62d262 Compare February 12, 2025 19:51

pujal0909 and others added 2 commits February 12, 2025 14:53

Merge branch 'master' into truncate-lrecl-target

3c2313c

Signed-off-by: Pujal Gandhi <71276682+pujal0909@users.noreply.github.com>

removed unused var

61be69b

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-advanced-security bot found potential problems Feb 12, 2025

View reviewed changes

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show resolved Hide resolved

fixed unit tests

a1714cb

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-advanced-security bot found potential problems Feb 13, 2025

View reviewed changes

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

pujal0909 added 2 commits February 12, 2025 21:27

updated integration snapshots

689f61e

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

updated snapshots

2161e6e

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

pujal0909 marked this pull request as ready for review February 13, 2025 13:24

pujal0909 requested review from t1m0thyj, gejohnston, zFernand0, awharn, traeok, ATorrise, adam-wolfe and jace-roell February 13, 2025 13:24

gejohnston approved these changes Feb 18, 2025

View reviewed changes

packages/zosfiles/src/methods/copy/Copy.ts Fixed Show resolved Hide resolved

updated unit tests

9d8dcce

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

t1m0thyj reviewed Feb 21, 2025

View reviewed changes

packages/cli/CHANGELOG.md Outdated Show resolved Hide resolved

updated changelogs

deca9fd

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

pujal0909 requested review from ATorrise and t1m0thyj February 21, 2025 14:52

This comment was marked as outdated.

Sign in to view

traeok self-requested a review February 21, 2025 15:37

traeok approved these changes Feb 21, 2025

View reviewed changes

zFernand0 approved these changes Feb 21, 2025

View reviewed changes

added progress bars

02d6c35

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-advanced-security bot found potential problems Feb 24, 2025

View reviewed changes

updated unit tests

4a3bcaa

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

pujal0909 requested a review from anaxceron February 25, 2025 02:56

github-advanced-security bot found potential problems Feb 25, 2025

View reviewed changes

packages/cli/src/zosfiles/copy/ds/Ds.handler.ts Fixed Show fixed Hide fixed

removed unused imports

2204550

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

updated code coverage

ca27f0f

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

anaxceron reviewed Feb 25, 2025

View reviewed changes

packages/cli/CHANGELOG.md Outdated Show resolved Hide resolved

packages/zosfiles/CHANGELOG.md Outdated Show resolved Hide resolved

updated changelogs - PR comments

f9436b5

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

github-advanced-security bot found potential problems Feb 25, 2025

View reviewed changes

packages/zosfiles/__tests__/__unit__/methods/copy/Copy.unit.test.ts Fixed Show fixed Hide fixed

anaxceron approved these changes Feb 25, 2025

View reviewed changes

small fix

e35a30d

Signed-off-by: Pujal <pujal.gandhi@broadcom.com>

zFernand0 mentioned this pull request Feb 26, 2025

copyPds: Misssing source PDS throws a non-user-friendly error #2447

Open

zFernand0 merged commit 16b4a25 into master Feb 26, 2025
25 checks passed

zFernand0 deleted the truncate-lrecl-target branch February 26, 2025 16:18

zFernand0 added the release-current Indicates that there is no new functionality being delivered label Feb 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record lines truncation of target PDS Enhancement #2433

Record lines truncation of target PDS Enhancement #2433

pujal0909 commented Feb 9, 2025 •

edited

Loading

codecov bot commented Feb 9, 2025 •

edited

Loading

github-actions bot commented Feb 13, 2025

This comment was marked as outdated.

traeok left a comment

zFernand0 left a comment

zFernand0 Feb 26, 2025

zFernand0 Feb 26, 2025

github-actions bot commented Feb 25, 2025

anaxceron left a comment

anaxceron left a comment

sonarqubecloud bot commented Feb 25, 2025

zFernand0 commented Feb 26, 2025

Record lines truncation of target PDS Enhancement #2433

Record lines truncation of target PDS Enhancement #2433

Conversation

pujal0909 commented Feb 9, 2025 • edited Loading

codecov bot commented Feb 9, 2025 • edited Loading

Codecov Report

github-actions bot commented Feb 13, 2025

This comment was marked as outdated.

traeok left a comment

Choose a reason for hiding this comment

zFernand0 left a comment

Choose a reason for hiding this comment

LGTM! 😋

zFernand0 Feb 26, 2025

Choose a reason for hiding this comment

zFernand0 Feb 26, 2025

Choose a reason for hiding this comment

github-actions bot commented Feb 25, 2025

anaxceron left a comment

Choose a reason for hiding this comment

anaxceron left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 25, 2025

Quality Gate passed

zFernand0 commented Feb 26, 2025

pujal0909 commented Feb 9, 2025 •

edited

Loading

codecov bot commented Feb 9, 2025 •

edited

Loading