Need to make correction for opening file larger than 2 GB

[edwardhartnett]

Need to make correction for opening file larger than 2 GB.

[BoiVuong-NOAA]

Hi Ed,
I am unable to solve in G2 library for opening file larger than 2 GB. Please re-assign to someone in nceplibs team.

[edwardhartnett]

The test code should create a file larger than 2 GB and then try to open it.

[Hang-Lei-NOAA]

This has been addressed on v3.4.2 and corrected on v3.4.3.
Okay to close.

[kgerheiser]

Hang this was not related to v3.4.2 which was for opening file units > 100.

[kgerheiser]

What I've gathered is that there is an integer that's overflowing somewhere and it needs to be changed to 64-bit. Gfortran has the flag -ftrapv which is supposed to catch integer overflow. g2 could be compiled with that and then run a > 2GB file through g2 to see where the overflow occurs.

[kgerheiser]

NCEPLIBS-g2/src/skgb.f

Line 32 in e184a5d

SUBROUTINE SKGB(LUGB,ISEEK,MSEEK,LSKIP,LGRIB)

I was able to get DEGRIB2 to work after changing the integers here to int64. There's more, and anything that deals with byte counts needs to be made int64.

[MatthewPyle-NOAA]

I'm also running into issues reading > 2 GB GRIB2 files, so happy to see that this issue is being addressed. Is there a timeline for version 3.5.0?

[GeorgeGayno-NOAA]

I am also having problems with > 2GB files. That function must work to support this ufs_utils project - ufs-community/UFS_UTILS#660. That is a high priority project according to @EricRogers-NOAA.

[edwardhartnett]

@kgerheiser so in file getg2ir.F90 did you make these variables:
lugb, iseek, msk1, lskip, lgrib

into INTEGER(KIND=8)? Or did you turn all the integers into KIND=8?

[kgerheiser]

I'm trying this again with @GeorgeGayno-NOAA's test case.

[GeorgeGayno-NOAA]

I'm trying this again with @GeorgeGayno-NOAA's test case.

Let me add some diagnostic print so you can see where it breaks.

[kgerheiser]

I ran chgres_cube with the GNU -ftrapv flag that catches integer overflow and this is the backtrace:

Backtrace for this error:
#0  0x2b30e327d3ff in ???
#1  0x2b30e327d387 in ???
#2  0x2b30e327ea77 in ???
#3  0x2b30e2dead88 in __addvsi3
	at /tmp/Role.Apps/spack-stage/spack-stage-gcc-9.2.0-wqdecm4rkyyhejagxwmnabt6lscgm45d/spack-src/libgcc/libgcc2.c:92
#4  0x61ce5e in skgb_
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/NCEPLIBS-g2/src/skgb.F90:53
#5  0x61b7b1 in getg2ir_
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/NCEPLIBS-g2/src/getg2ir.F90:131
#6  0x612223 in getidx_
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/NCEPLIBS-g2/src/getidx.F90:126
#7  0x610c35 in getgb2_
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/NCEPLIBS-g2/src/getgb2.F90:168
#8  0x57c0ce in define_input_grid_grib2
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/UFS_UTILS/sorc/chgres_cube.fd/model_grid.F90:672
#9  0x4cb7d6 in chgres
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/UFS_UTILS/sorc/chgres_cube.fd/chgres.F90:90
#10  0x4c8c2c in main
	at /scratch1/NCEPDEV/nems/Kyle.Gerheiser/ufs-utils-test/UFS_UTILS/sorc/chgres_cube.fd/chgres.F90:26

Which is this call to baread:

NCEPLIBS-g2/src/skgb.F90

Line 53 in a8bdf75

CALL BAREAD(LUGB,KS+K+KG-4,4,K4,Z4)

And the addition of KS+K+KG-4 which is the number of bytes to skip.

So, this will also require a change in bacio as well. And anyone that uses these functions will also have to change to match the int64 interface. You can't just pass an int32 to an int64 function because the data will be offset and you'll get junk. (Or can you because the bottom 32 bits are the same?)

[edwardhartnett]

What if we add new functions to bacio which handle 64-bit?

[kgerheiser]

Yes, maybe that is a solution. And the 32-bit interface could cast to in64 internally.

[edwardhartnett]

OK, I am looking at bacio. Actually there already is a function bareadl() which handles 64-bit ints:

SUBROUTINE BAREADL(LU, IB, NB, KA, A)
  USE BACIO_MODULE
  IMPLICIT NONE
  INTEGER, intent(in) :: LU
  INTEGER(kind=8), intent(in) :: IB,NB
  INTEGER(kind=8), intent(out) :: KA
  CHARACTER, intent(out) :: A(NB)

So if you just change your call in skgb.F90 to use bareadl() you can pass in your 64-bit arguments.

[kgerheiser]

Thanks @edwardhartnett

[edwardhartnett]

OK, unfortunately this is something Kyle didn't get to finish. I'm taking a look at this now,

@GeorgeGayno-NOAA is there a test file I can use >2 GB on hera?

[GeorgeGayno-NOAA]

OK, unfortunately this is something Kyle didn't get to finish. I'm taking a look at this now,

@GeorgeGayno-NOAA is there a test file I can use >2 GB on hera?

You can try: /scratch1/NCEPDEV/da/George.Gayno/noscrub/eric.rrfs/fv3lam.t00z.prslev.f003.grib2

[edwardhartnett]

OK, I have taken a stab at this and I think some major surgery will be required. The problem is that the skgb() subroutine takes integer parameters (so *4 on _4 builds, and 8 on _8 builds) and it needs to take integer8 always.

So this means I need to add a new function skgb8(), which does the same thing as skgb() but with I*8 parameters. Then I can combine these two functions so it is really just one function doing the work.

[GeorgeGayno-NOAA]

Thanks for fixing this. We can now resume work on ufs-community/UFS_UTILS#660

Has the latest G2 tag been installed on any of our machines?

[edwardhartnett]

No but it's coming soon, along with a new release of grib_util.

[GeorgeGayno-NOAA]

No but it's coming soon, along with a new release of grib_util.

Great. @LarissaReames-NOAA would like to use it on Jet.