Chronologer Results are Not Repeatable for Moderately Sized Inputs

[nwamsley1]

Hello, I've found Koina very useful for fragment intensity predictions, and now I am trying to use the Chronologer model via cURL. However, there is a bug.

For a moderately sized input of say, 5 sequences, I seem get the same result every time, and the numbers are sensible. For example:

curl -s https://koina.wilhelmlab.org:443/v2/models/Chronologer_RT/infer -d '{"id":"0","inputs":[{"name":"peptide_sequences","shape":[5,1],"data":["AEVTPSQHGNR","AEVTPSQHGNR","AEVTPSQHGNRTFSYTLEDHTK","AEVTPSQHGNRTFSYTLEDHTK","AM[UNIMOD:35]FTNGLR"],"datatype":"BYTES"}]}

I get the following ouput when I run the command.

"{\"id\":\"0\",\"model_name\":\"Chronologer_RT\",\"model_version\":\"1\",\"parameters\":{\"sequence_id\":0,\"sequence_start\":false,\"sequence_end\":false},\"outputs\":[{\"name\":\"rt\",\"datatype\":\"FP32\",\"shape\":[5,1],\"data\":[1.3737053871154786,1.3737053871154786,6.873213768005371,6.873213768005371,6.482181549072266]}]}"

However, if I try to submit 100 sequences, the result varies each time I submit the job. Moreover, the predicted retention times lie far outside the reasonable range. For example, -5637.58. I do not get any error and my input is formatted the same as in the smaller case. I've copied an example at the bottom of this post.

Here is the command:

curl -s https://koina.wilhelmlab.org:443/v2/models/Chronologer_RT/infer -d '{"id":"0","inputs":[{"name":"peptide_sequences","shape":[100,1],"data":["AEVTPSQHGNR","AEVTPSQHGNR","AEVTPSQHGNRTFSYTLEDHTK","AEVTPSQHGNRTFSYTLEDHTK","AM[UNIMOD:35]FTNGLR","AMFTNGLR","AM[UNIMOD:35]FTNGLREQGMEVVSIEGIHPK","AMFTNGLREQGM[UNIMOD:35]EVVSIEGIHPK","AMFTNGLREQGMEVVSIEGIHPK","AM[UNIMOD:35]FTNGLREQGMEVVSIEGIHPK","AMFTNGLREQGM[UNIMOD:35]EVVSIEGIHPK","AMFTNGLREQGMEVVSIEGIHPK","AREYIYM[UNIMOD:35]HFGEVAK","AREYIYMHFGEVAK","AREYIYM[UNIMOD:35]HFGEVAK","AREYIYMHFGEVAK","AREYIYM[UNIMOD:35]HFGEVAK","AREYIYMHFGEVAK","AVEGFHM[UNIMOD:35]YIYEK","AVEGFHMYIYEK","AVEGFHM[UNIMOD:35]YIYEK","AVEGFHMYIYEK","AVEGFHM[UNIMOD:35]YIYERAK","AVEGFHMYIYERAK","AVEGFHM[UNIMOD:35]YIYERAK","AVEGFHMYIYERAK","AVEGFHM[UNIMOD:35]YIYERAK","AVEGFHMYIYERAK","AVRC[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQK","AVRC[UNIMOD:4]HSLTPNFLQMQLQK","AVRC[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQK","AVRC[UNIMOD:4]HSLTPNFLQMQLQK","C[UNIMOD:4]C[UNIMOD:4]AGAGSPR","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQM[UNIMOD:35]R","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQMR","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQM[UNIMOD:35]R","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQMR","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQM[UNIMOD:35]R","C[UNIMOD:4]C[UNIMOD:4]AGAGSPRPDPQMR","C[UNIMOD:4]EILQSDSR","C[UNIMOD:4]EILQSDSRC[UNIMOD:4]K","C[UNIMOD:4]EILQSDSRC[UNIMOD:4]K","C[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVK","C[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVK","C[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVK","C[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVKYDC[UNIMOD:4]EQR","C[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVKYDC[UNIMOD:4]EQR","C[UNIMOD:4]ETSAYM[UNIMOD:35]VADGAGEPC[UNIMOD:4]QSQLPLFK","C[UNIMOD:4]ETSAYMVADGAGEPC[UNIMOD:4]QSQLPLFK","C[UNIMOD:4]ETSAYM[UNIMOD:35]VADGAGEPC[UNIMOD:4]QSQLPLFK","C[UNIMOD:4]ETSAYMVADGAGEPC[UNIMOD:4]QSQLPLFK","C[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQK","C[UNIMOD:4]HSLTPNFLQMQLQK","C[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQK","C[UNIMOD:4]HSLTPNFLQMQLQK","C[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQK","C[UNIMOD:4]HSLTPNFLQMQLQK","C[UNIMOD:4]HSLTPNFLQMQLQKC[UNIMOD:4]EILQSDSR","C[UNIMOD:4]HSLTPNFLQM[UNIMOD:35]QLQKC[UNIMOD:4]EILQSDSR","C[UNIMOD:4]HSLTPNFLQMQLQKC[UNIMOD:4]EILQSDSR","C[UNIMOD:4]KDYLVK","C[UNIMOD:4]PEM[UNIMOD:35]TVAVGVGSR","C[UNIMOD:4]PEMTVAVGVGSR","C[UNIMOD:4]PEM[UNIMOD:35]TVAVGVGSR","C[UNIMOD:4]PEMTVAVGVGSR","C[UNIMOD:4]PEM[UNIMOD:35]TVAVGVGSRGSTMR","C[UNIMOD:4]PEMTVAVGVGSRGSTM[UNIMOD:35]R","C[UNIMOD:4]PEMTVAVGVGSRGSTMR","C[UNIMOD:4]PEM[UNIMOD:35]TVAVGVGSRGSTMR","C[UNIMOD:4]PEMTVAVGVGSRGSTM[UNIMOD:35]R","C[UNIMOD:4]PEMTVAVGVGSRGSTMR","C[UNIMOD:4]PEM[UNIMOD:35]TVAVGVGSRGSTMR","C[UNIMOD:4]PEMTVAVGVGSRGSTM[UNIMOD:35]R","C[UNIMOD:4]PEMTVAVGVGSRGSTMR","C[UNIMOD:4]PM[UNIMOD:35]VQTPKHLTLEEFIR","C[UNIMOD:4]PMVQTPKHLTLEEFIR","C[UNIMOD:4]PM[UNIMOD:35]VQTPKHLTLEEFIR","C[UNIMOD:4]PMVQTPKHLTLEEFIR","C[UNIMOD:4]PM[UNIMOD:35]VQTPR","C[UNIMOD:4]PMVQTPR","C[UNIMOD:4]RSDSQLIEC[UNIMOD:4]K","C[UNIMOD:4]RSDSQLIEC[UNIMOD:4]K","C[UNIMOD:4]VLHVM[UNIMOD:35]NGAVMYQIDSVVR","C[UNIMOD:4]VLHVMNGAVM[UNIMOD:35]YQIDSVVR","C[UNIMOD:4]VLHVMNGAVMYQIDSVVR","C[UNIMOD:4]VLHVM[UNIMOD:35]NGAVMYQIDSVVR","C[UNIMOD:4]VLHVMNGAVM[UNIMOD:35]YQIDSVVR","C[UNIMOD:4]VLHVMNGAVMYQIDSVVR","DDLNVRC[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVK","DDLNVRC[UNIMOD:4]ESEVFHAC[UNIMOD:4]INWVK","DEWHLVAPM[UNIMOD:35]LTR","DEWHLVAPMLTR","DEWHLVAPM[UNIMOD:35]LTR","DEWHLVAPMLTR","DEWHLVAPM[UNIMOD:35]LTRR","DEWHLVAPMLTRR","DEWHLVAPM[UNIMOD:35]LTRR","DEWHLVAPMLTRR","DEWHLVAPM[UNIMOD:35]LTRR","DEWHLVAPMLTRR"],"datatype":"BYTES"}]}'

And here are two outputs from the command requested back-to-back:
1)

julia> read(cmd, String)
"{\"id\":\"0\",\"model_name\":\"Chronologer_RT\",\"model_version\":\"1\",\"parameters\":{\"sequence_id\":0,\"sequence_start\":false,\"sequence_end\":false},\"outputs\":[{\"name\":\"rt\",\"datatype\":\"FP32\",\"shape\":[100,1],\"data\":[1.7800806760787964,1.3737053871154786,6.904702186584473,2.3537425994873049,6.489931106567383,173.1466827392578,265.48846435546877,-62.745689392089847,171.1584014892578,13.731771469116211,-257.09393310546877,15.239262580871582,8.325523376464844,-351.1920166015625,919.9856567382813,13.133613586425782,8.320616722106934,10.887491226196289,9.895777702331543,12.19365406036377,9.277403831481934,12.181988716125489,6.8597917556762699,-174.59075927734376,8.248983383178711,10.659605026245118,8.248983383178711,11.117879867553711,-658.2821044921875,12.935224533081055,20.59821128845215,-216.08863830566407,1.5738961696624756,-1375.6229248046876,-2281.731201171875,3.290607452392578,3.615488052368164,2.1271915435791017,-2114.36669921875,5.479742050170898,418.5933837890625,3.889080762863159,13.00690746307373,-108.83185577392578,13.832070350646973,8.809208869934082,13.05402660369873,15.513964653015137,9805.345703125,15.516176223754883,16.5393123626709,12.837430953979493,14.710951805114746,15.567415237426758,13.909455299377442,4.334645748138428,14.695572853088379,14.941089630126954,978.8344116210938,15.684898376464844,3.862645149230957,6.978960990905762,10.550773620605469,7.13844108581543,10.330988883972168,6.735597610473633,7.981585502624512,12.179962158203125,6.705447196960449,17.233427047729493,9.329728126525879,7.518546104431152,8.975935935974121,9.662302017211914,14.234349250793457,15.100383758544922,13.537030220031739,14.36449146270752,3.529456377029419,6.8434672355651859,4.15437650680542,-54.94418716430664,-99.99796295166016,15.20481014251709,-298.7019348144531,15.594488143920899,15.894510269165039,17.749530792236329,2381.394287109375,15.583329200744629,14.36528205871582,16.45985221862793,14.372495651245118,16.47905731201172,12.172661781311036,14.131251335144043,12.272510528564454,14.47985553741455,10.456217765808106,14.233207702636719]}]}"

julia> read(cmd, String)
"{\"id\":\"0\",\"model_name\":\"Chronologer_RT\",\"model_version\":\"1\",\"parameters\":{\"sequence_id\":0,\"sequence_start\":false,\"sequence_end\":false},\"outputs\":[{\"name\":\"rt\",\"datatype\":\"FP32\",\"shape\":[100,1],\"data\":[-1.8220674991607667,-4.4370269775390629,357.0726318359375,22.577287673950197,6.551889419555664,10.07076644897461,19.71158218383789,14.554352760314942,13.951148986816407,13.794878959655762,13.743903160095215,15.571014404296875,9.287910461425782,10.949015617370606,5229.52392578125,11.262345314025879,6.735544204711914,10.932809829711914,9.625011444091797,12.17946720123291,9.308941841125489,32.49226760864258,155.28070068359376,2208.198974609375,8.319160461425782,10.719996452331543,8.201605796813965,2633.66650390625,11.339364051818848,12.85756778717041,11.339364051818848,13.037720680236817,148.86209106445313,1188.2615966796876,3.7069058418273927,1.6351423263549805,22.852502822875978,6.2225422859191898,2503.68505859375,13.91500473022461,5.205883026123047,4.127349853515625,13.525403022766114,10.215897560119629,15.712485313415528,13.065698623657227,13.065343856811524,8.721813201904297,22.536861419677736,15.516176223754883,13.358513832092286,0.33578014373779299,12.024467468261719,13.03929328918457,21.97815704345703,19.092304229736329,15.514303207397461,15.472105026245118,14.411073684692383,7.82567024230957,3.994070053100586,7.217626571655273,21.507030487060548,8.09695816040039,10.02344036102295,6.688414573669434,9.708439826965332,22.66831398010254,8.9180269241333,7.739884376525879,8.552244186401368,5.460411548614502,8.332500457763672,9.45807933807373,12.512453079223633,15.359905242919922,12.84699535369873,7.724985122680664,3.621309518814087,6.096827507019043,9.331334114074707,21.527860641479493,25.07730484008789,5.929878234863281,23.0067195892334,-1318.0037841796876,23.039833068847658,589.5706176757813,14.431703567504883,22.44483184814453,15.154211044311524,-653.9866333007813,8.060250282287598,24.589574813842775,12.788991928100586,12.668136596679688,17.410511016845704,28.94529151916504,13.280466079711914,14.198739051818848]}]}"

In both cases the results are different, and there are some extreme negative values in the output.

[LLautenbacher]

Hi, thanks for reaching out! I fixed this temporarily by decreasing the batch size to reproducible levels while we looked into the underlying cause.