Fix participants age for tokyoSkyra site #96

alexfoias · 2021-09-27T20:26:01Z

Updated the information based on the information from our collaborators.

I tried to trace back when did the issue appear, but it was impossible due to the various changes in infrastructure.

jcohenadad · 2021-09-27T20:34:26Z

I tried to trace back when did the issue appear, but it was impossible due to the various changes in infrastructure.

you need to trace it back because we need to make sure:

the participant IDs did not change (ie: the info given by Kouhei recently are valid for this BIDS dataset)
this error did not happen for other datasets

so please, go back in time, look for emails, etc. document as much as possible. This is a very serious issue and we have a responsibility to not disseminate fake results.

jcohenadad · 2021-09-27T20:58:33Z

this error did not happen for other datasets

e.g.: the tokyoIngenia values are also suspicious:

sub-tokyoIngenia01	M	25	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia02	F	32	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia03	M	36	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia04	F	29	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia05	F	28	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia06	M	24	-	-	2019-06-13	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"
sub-tokyoIngenia07	M	35	-	-	2019-10-01	tokyo	Ingenia the University of Tokyo	Philips	Ingenia	-	5.3.1_5.3.1.1	"K. Kamiya, Y. Suzuki"

alexfoias · 2021-09-29T18:49:37Z

I checked in all my emails. I could only find demographics for Juntendo.

I think we received the data already converted to nifti. In the email conversation with @jcohenadad with the subject Re: Spinal cord generic protocol : consent from participants @ Tokyo Univ sent on Wed, Apr 3, 2019 at 1:40 PM, I asked if we had the raw data, but I didn't get an answer.

In the changes files, we don't have all the history of changes (the one from openeuro).

I don't know who added the data to the dataset.

alexfoias · 2021-09-29T18:53:11Z

This error seemed to be in the initial commit on openneuro https://github.com/OpenNeuroDatasets/ds002902

jcohenadad · 2021-09-29T19:54:50Z

I checked in all my emails. I could only find demographics for Juntendo.

I think we received the data already converted to nifti. In the email conversation with @jcohenadad with the subject Re: Spinal cord generic protocol : consent from participants @ Tokyo Univ sent on Wed, Apr 3, 2019 at 1:40 PM, I asked if we had the raw data, but I didn't get an answer.

sorry for the oversight-- the answer is: no, we don't have the raw data

In the changes files, we don't have all the history of changes (the one from openeuro).

I don't know who added the data to the dataset.

me neither

jcohenadad · 2021-09-29T19:57:57Z

OK, we know these are the same subjects. So the last sanity check is to verify that the sub01 on tokyoIngenia corresponds to sub01 on tokyoSkyra, etc. @alexfoias can you please do that (look at the T1w MRIs and you will easily find out if these are the same subjects)

alexfoias · 2021-09-30T19:07:13Z

I did a visual checkout and the subjects seem to be correctly identified (similar anatomy for all 3 centers).

kousu

Because

we have a responsibility to not disseminate fake results.

this entire discussion needs to be summarized into the commit message before this is merged. Maybe even expanded, not summarized: include email addresses, institution names, URLs, dates, explanations, everything to explain where the error came from, how it was resolved.

That's the big benefit of using git diligently -- so that future us can trace back what happened to a dataset, and why.

kousu · 2021-11-15T06:30:26Z

Hey, I fixed CI, so now we can actually talk about reviewing this properly.

What is "the information from our collaborators" that triggered this update? Who are they? Names and academic email addresses?
The tokyoIngenia values also need to be synchronized.
Should we delete https://openneuro.org/datasets/ds002902? Since it's wrong?
a. If we do delete it, how do we make sure it's also deleted from https://github.com/OpenNeuroDatasets/ds002902/? I know their Github syncer is pretty flakey sometimes.
Let's make sure to write a relatively detailed commit message justifying this change.

Also, this is probably worth a different issue, but:

Should we rearrange the dataset to put all the data from the same physical people under the same (e.g. sub-tokyo*01 -> sub-tokyo01, sub-vuiis*04 -> sub-vuiss04) and represent the scanner used as a tag in the filenames?

jcohenadad · 2021-11-15T14:41:22Z

Should we delete https://openneuro.org/datasets/ds002902? Since it's wrong?

yes ( i thought we already did it @alexfoias ?)

Should we rearrange the dataset to put all the data from the same physical people under the same

sorry i don't understand your suggestion-- what do you mean by "data from the same physical people"?

alexfoias · 2021-11-15T16:03:42Z

I have removed all datasets from OpenNeuro. (This screenshot is take from the generic account used to create the datasets.)

.

kousu · 2021-11-15T16:14:04Z

I have removed all datasets from OpenNeuro.

🤦 you're right. Okay, punting: OpenNeuroOrg/openneuro#2484

(This screenshot is take from the generic account used to create the datasets.)

By the way does anyone else have that credential? Does Julien? Should I? Better yet, can we make a Team on openneuro and all have our own credentials instead? Or are we giving up on OpenNeuro entirely?

kousu · 2021-11-16T14:57:49Z

Should we delete https://openneuro.org/datasets/ds002902? Since it's wrong?

yes ( i thought we already did it @alexfoias ?)

The older version is off OpenNeuro for good now: OpenNeuroOrg/openneuro#2484

Should we rearrange the dataset to put all the data from the same physical people under the same

sorry i don't understand your suggestion-- what do you mean by "data from the same physical people"?

-> #102

2. The `tokyoIngenia` values also need to be synchronized.

And all tokyo750w* too!: 10c5e45

Still TODO: writing a good commit message.

What is "the information from our collaborators" that triggered this update? Who are they? Names and academic email addresses?

Let's make sure to write a relatively detailed commit message justifying this change.

@alexfoias could we see the email that inspired this PR?

kousu · 2021-11-16T15:31:11Z

New CI bug? https://github.com/spine-generic/data-multi-subject/runs/4226501836?check_suite_focus=true#step:11:2598

 (recording state in git...)
git-annex: get: 3 failed
Error: Process completed with exit code 1.

Looks like maybe it's Amazon S3 having a minor outage?

https://github.com/spine-generic/data-multi-subject/runs/4226501836?check_suite_focus=true#step:11:620

get derivatives/labels/sub-stanford04/anat/sub-stanford04_acq-T1w_MTS_seg-manual.nii.gz (from amazon...) 
  download failed: Internal Server Error

  Unable to access these remotes: amazon

  Try making some of these repositories available:
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
   	5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject
   	9e4d13f3-30e1-4a29-8b86-670879928606 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject

  (Note that these git remotes have annex-ignore set: origin)
failed

https://github.com/spine-generic/data-multi-subject/runs/4226501836?check_suite_focus=true#step:11:658

get derivatives/labels/sub-tehranS01/anat/sub-tehranS01_T1w_labels-disc-manual.nii.gz (from amazon...) 
  download failed: Internal Server Error

  Unable to access these remotes: amazon

  Try making some of these repositories available:
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
   	5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject
   	9e4d13f3-30e1-4a29-8b86-670879928606 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject

  (Note that these git remotes have annex-ignore set: origin)
failed

https://github.com/spine-generic/data-multi-subject/runs/4226501836?check_suite_focus=true#step:11:711

get derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_T2star_rms_gmseg-manual.nii.gz (from amazon...) 
  download failed: Internal Server Error

  Unable to access these remotes: amazon

  Try making some of these repositories available:
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
   	5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- julien@julien-macbook.local:~/code/spine-generic/data-multi-subject
   	9e4d13f3-30e1-4a29-8b86-670879928606 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject

  (Note that these git remotes have annex-ignore set: origin)
failed

The following run passed: https://github.com/spine-generic/data-multi-subject/runs/4226604983?check_suite_focus=true

alexfoias · 2021-11-17T21:11:23Z

@kousu Here is the info from the email:

Subject01 M 36 177cm 70kg
Subject02 M 36 168cm 80kg
Subject03 M 32 162cm 57kg
Subject04 F 34 161cm 65kg
Subject05 F 35 154cm 51kg
Subject06 F 30 160cm 48kg
Subject07 M 29 165cm 60kg

jcohenadad · 2022-01-19T16:51:22Z

@alexfoias can you please update this PR after #112

alexfoias

The values are the ones transferred by the collaborator.

participants.tsv

fix info

d7551a0

alexfoias requested a review from jcohenadad September 27, 2021 20:26

Merge branch 'master' into af/fix_participants

1c17364

jcohenadad approved these changes Nov 10, 2021

View reviewed changes

jcohenadad changed the title ~~Fix participants~~ Fix participants age for tokyoSkyra site Nov 10, 2021

jcohenadad added this to the next-release milestone Nov 10, 2021

kousu suggested changes Nov 13, 2021

View reviewed changes

This was referenced Nov 15, 2021

Sync deletion OpenNeuroOrg/openneuro#2484

Closed

Deal with subjects scanned on different scanners #102

Open

kousu added 2 commits November 16, 2021 10:06

Merge branch 'master' into af/fix_participants

458d65f

Update remaining tokyo demographic data.

10c5e45

alexfoias and others added 2 commits January 19, 2022 15:57

fix participants

36b71fc

Merge branch 'master' into af/fix_participants

63a4ece

alexfoias requested review from jcohenadad and kousu January 19, 2022 20:59

alexfoias removed the request for review from kousu January 19, 2022 20:59

alexfoias commented Jan 19, 2022

View reviewed changes

Merge branch 'master' into af/fix_participants

bba70c0

jcohenadad reviewed Jan 20, 2022

View reviewed changes

participants.tsv Outdated Show resolved Hide resolved

Update participants.tsv

f927efc

jcohenadad approved these changes Jan 20, 2022

View reviewed changes

jcohenadad merged commit 4e15999 into master Jan 20, 2022

jcohenadad deleted the af/fix_participants branch January 20, 2022 01:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix participants age for tokyoSkyra site #96

Fix participants age for tokyoSkyra site #96

alexfoias commented Sep 27, 2021

jcohenadad commented Sep 27, 2021

jcohenadad commented Sep 27, 2021

alexfoias commented Sep 29, 2021

alexfoias commented Sep 29, 2021

jcohenadad commented Sep 29, 2021

jcohenadad commented Sep 29, 2021

alexfoias commented Sep 30, 2021

kousu left a comment •

edited

Loading

kousu commented Nov 15, 2021 •

edited

Loading

jcohenadad commented Nov 15, 2021

alexfoias commented Nov 15, 2021

kousu commented Nov 15, 2021

kousu commented Nov 16, 2021 •

edited

Loading

kousu commented Nov 16, 2021 •

edited

Loading

alexfoias commented Nov 17, 2021

jcohenadad commented Jan 19, 2022

alexfoias left a comment

Fix participants age for tokyoSkyra site #96

Fix participants age for tokyoSkyra site #96

Conversation

alexfoias commented Sep 27, 2021

jcohenadad commented Sep 27, 2021

jcohenadad commented Sep 27, 2021

alexfoias commented Sep 29, 2021

alexfoias commented Sep 29, 2021

jcohenadad commented Sep 29, 2021

jcohenadad commented Sep 29, 2021

alexfoias commented Sep 30, 2021

kousu left a comment • edited Loading

Choose a reason for hiding this comment

kousu commented Nov 15, 2021 • edited Loading

jcohenadad commented Nov 15, 2021

alexfoias commented Nov 15, 2021

kousu commented Nov 15, 2021

kousu commented Nov 16, 2021 • edited Loading

kousu commented Nov 16, 2021 • edited Loading

alexfoias commented Nov 17, 2021

jcohenadad commented Jan 19, 2022

alexfoias left a comment

Choose a reason for hiding this comment

kousu left a comment •

edited

Loading

kousu commented Nov 15, 2021 •

edited

Loading

kousu commented Nov 16, 2021 •

edited

Loading

kousu commented Nov 16, 2021 •

edited

Loading