Dear Dr. Ram,
Thank you very much for your review of our paper and that by Dr. Carl Boettiger. All of the suggestions were incredibly helpful and we integrated changes in response to most of them. Detailed point by point responses are included below. We hope that you will find the current version of the ms suitable for publication in Ideas in Ecology and Evolution.
Regards, Ethan White
1. Clarify the target audience
Although the title of the article is quite broad and some guidelines are general enough to be applicable to any research community, the article is clearly geared towards environmental scientists. More specifically, the repositories suggested in the article are all associated with EEB (and closely related) journals and communities. Also the guidelines for preparing data are also limited to basic tabular data (and do not cover other heterogeneous and large data types which are characteristic of communities such as astronomy and physics). I don't make the last point as a criticism or to suggest that the review is not comprehensive. Instead I'm suggesting that clarifying the audience both in the title and early in the introduction could provide some additional focus.
We have clarified the target audience in the Introduction by highlighting the fact that this paper is intended to be a very simple introduction to these ideas and that our examples are targeted at EEB folks (but still apply more broadly). We didn't change the title because we feel that the application of these ideas is quite broad (e.g., most discipline uses tables) and because the journal that it is being published in should provide additional EEB context.
2. logistical versus technical?
When you say logistical do you actually mean technical? A vast majority of researchers still use GUI tools and manipulate most of their data by hand. This subset of researchers also lack the technical skills necessary to prepare and submit their data to a repository. So to me this seems more of a technical than (a bit of both really) a logistical challenge.
Changed.
3. Sections 3-5:
I recommend skimming Michener and Jones 2012 for additional points to share in this context (see citation at the bottom).
A great paper that definitely needed to be cited. Thanks!
Steps 3, 4 and parts of 5 from the review have several additional useful suggestions for someone preparing their data for sharing purposes. In particular, it would be worth noting that:
a) Storing data in relational databases can better ensure that multiple data types don't get mixed up in the same column (which as you point out is much more common problem in programs like Excel).
We have added a general mention of this idea, but avoided getting into the specifics of DBMS to keep the paper simple and easily accessible to a broad audience.
b) The Michener/Jones section on metadata (also see the part titled assure) contains some additional information that also applies to the QA/QC section.
We have added several citations in this section and added a new section to the conclusion emphasizing their ideas about the benefits of planning for data management in advance.
Given that ecological experiments rarely go exactly as planned, and data can be messy, researchers should strive to describe the circumstances as accurately as possible so future consumers can best decide if the data can be integrated into a new study. The most common scenario is that detailed metadata are missing which means that a careful downstream consumer will have to discard the data rather than use one they do not fully trust. This bit seems implied rather than explicitly pointed in the metadata section.
We have added additional language about data quality and the importance of metadata.
4. QA/QC
In addition to just giving the data a once over before sharing, there are a couple of other useful suggestions that might be worth sharing in this section:
Trusting someone else's data is often very hard for downstream consumers (There is some interesting discussion in Zimmerman 2008). So if people can provide additional flags or indicators about the data quality (not QA), that could help lower the barrier to reuse. Obviously this is suggestion may not apply to certain types of data.
Great idea. Added.
When taking about sanity checks, it might also be worth mentioning that the same steps could be done programmatically. For example, in R one can use melt and cast to actually figure out if there are any missing measurements without manually scanning spreadsheets. Mentioning this a second time (in addition to the R/Python Pandas mention) might actually help nudge readers towards better scientific computing practices (although any discussion on this matter is well outside the scope of this article).
We have added general language about this to the manuscript.
5. Citation suggestion
In section #8, it might be worth citing Schultheiss (2011) where they quantitatively show that data stored on lab computers and web pages disappear often.
As already discussed in the issue queue the citation isn't a good match since this paper is about web services rather than datasets.
6. figshare
Many of the repositories mentioned in this section are specific to certain data types (e.g. Genbank) or require a paper be associated with a publication in a member journal (e.g. Dryad). I noticed that although you mentioned figshare in that table you don't actually say that it's the easiest and fastest option available in table 2. This would be really helpful for readers in communities where there is no data sharing culture whatsoever (so they can't really follow their peers) or rely on institutional support (like what DataUP provides for the UC).
As much as we love figshare we thought it was best not to overemphasize a particular repository.
Figshare should be figshare. They do not capitalize their name.
Done.
1. "Share your data"
Motivate the data sharing more directly for the reader -- who benefits from these practices? (e.g. highlight individal benefits, community benefits may be more self-evident)
All references you cite identify a cultural challange as dominant. While addressing that is not really the scope / objective here, it would be worth acknowledging this. Recommendation #1 kind of addresses this, but cannot really do justice to it in two paragraphs. The paper will serve as a practical guide to those intersted in doing so, rather than convincing those that have doubts.
We have added some additional motivation in both this section and the Conclusions while still trying to maintain the focus of the current piece on practice rather than justification since Poisot et al. will handle that area more thoroughly.
That said, the topic sentence at line 50 probably shouldn't be "scientists are reluctant to share..", but something to the effect that incentives have previously been insufficient to encourage sharing but are rapidly shifting.
Great suggestion. Done.
L50 - 64 The structure of the arguments jump around a bit in this paragraph. I'd recommend something like "1. advantages/ reasons to share", "2. reasons scientists don't share so much yet" and "3. changes". You provide mention of changes in funding requirements and laws, only to switch back to the "reluctant to share".
We have restructured this section along these lines.
L. 49. Great set of links. In addition to FASTR, maybe link the recent whitehouse statement that would mandate this as well? Also, not sure what the journal policies are for linking vs formally citing this material.
Great idea. Done.
Lines 33:34 Jones et al are good references, but broader than the evidence for not following 'best-practices'. Consider these citations
Palmer M, Bernhardt ES, Chornesky EA, Collins SL, Dobson AP, et al. 2005. Eco- logical science and sustainability for the 21st century. Front. Ecol. Environ. 3:4–11
We read this very interesting paper, but it didn't seem like an approapriate citation in this context.
You cite this later, but might be appropriate to mention it here;
Parr CS, Cummings MP. 2005. Data sharing in ecology and evolution. Trends Ecol. Evol. 20(7):362–63
Done.
2. Metadata
L 82-84. Like much other advice, you casually throw out names of "metadata standards", some of which are defined as XML Schema, some of which are vocabularies or proper ontologies, etc., along side vague recommendations to "describe the data". My intuition is that the average ecologist reading this will go through these recommendations like this:
1) Describe the "What, when, where, how of the data" -- oh, that'll be in the published paper.
2) "How to access the data" email me, duh. I'm the corresponding author. It says so on the paper.
3) "Suitability of the data in answering other questions" Stuff I'll probably discuss in the introduction. If you don't know what it's suitable for, you probably shouldn't be using my data anyhow.
4) "Warnings about known problems" You kidding me? My data does not have problems or inconsistencies!
5) "Information to confirm that the data is properly imported, like the number of rows and columns". Ah ha! What a good idea, I'll list the number of rows and columns of my data and I'll be cutting edge.
We've already discussed this extensively with the reviewer in the issue in the GitHub repo, but for the sake of the journal process and to make the editors life easier we will reiterate our main points.
Our impression is that ecologists really do want to do these things better and will respond positively to the paper (in fact the initial response to the preprint has been very positive). In fact our impression is that the error is often getting too technical and asking for standardized machine readable metadata too early from the average ecologist and thus scaring them off from sharing their data or making meaningful steps to making it easier to work with.
If we are serious about improving ecological metadata, I think we need something more persuasive about how it can add value and to whom. The current manuscript makes no attempt to explain the value of machine readable standards (even merely to point out they are machine-readable). Yes, you point to three excellent tools which helping to lower the technical barrier, but not the social / motivational barriers.
We have added additional motivation and used the term machine readable.
Also, there's a lot of overlap here with the issues in #4 "Standard data formats", but the link is not made clear.
This is a challenging link to understand for beginners, but we have attempted to start to make that connection in Section 4.
3. Unprocessed form of data
I love this section. Just the other day I was so happy to see that this fascinating research I was reading had public data, and so dismayed to see that the raw time series I needed were not available. You might want some discussion of just what "raw data" means -- one person's raw data is another's highly processed data.
We have added more explicity language about what we mean by "raw data".
4. Standard data formats
Great section, with nice concrete recommendations that can easily be understood and implemented by anyone.
My only gripe is that a lot of the issues discussed here are addressed by the metadata standards you cite earlier, but the connection is completely ignored and probably lost on most readers.
We added a sentence at the end of the section to highlight the linkage between metadata and these recommendations.
Section 5 really feels like a subsection of 4.3 "standard formats within cells", but given the importance of the issue I'm happy to see it remain it's own section.
We agree. We spent a lot of time working on how to best split up the information contained in Sections 4 and 5. We couldn't figured out any perfect solutions, but this is the best compromise we've found.
6. Combining with other data sets
A good section in principle, but not very concrete. It sounds like your primary advice here is to avoid undefined abbreviations, and to include columns with generic information like species or lat/long coordinates that might be useful to others. In both cases, you appear to be citing issues that have more to do with metadata. E.g. if I collect all my data on a single species at a single geographic site; is it really necessary that I add columns for species and lat-long, rather than define this information in the metadata?
Good point. We made this section more explicit and addressed cases where this kind of information is more suitable as metadata.
I think more helpful here would be to emphasize the value of collecting/recording additional generic data even if it is not relevant for your study. (Researchers not interested in spatial or seasonal patterns do not always report spatial coordinates or sampling dates and times, temperature/weather information, gross measurements of sampled individuals like length and mass etc.)
We emphasized the value of reporting this kind of information if it was collected, but didn't go as far as suggesting that researchers collect data they don't personally need. While we agree with the value of collecting the additional data we didn't feel that it would play well to ask over-stretched field researchers to collect additional data to make the lives of synthetic folks easier.
7. Quality control
Very good section, but you ignore any mention of tools that can assist with at least some of these things, from the very basic (e.g. reading the file into software such as R and performing basic visual inspection / graphing to make sure it is imported) to richer options possible with stricter formats (XML schema validation, etc.).
We have added general language about the potential for automated quality control.
8. Repositories
Great section. Emphasize the personal advantages here? Perhaps with references that have demonstrated the personal benefits (citation, ease of re-use / avoiding file loss, etc.)
We are unaware of references for personal benefits of established repositories as opposed to the general benefits of data sharing.
Could you consider mentioning archiving things like an R script that is used to clean / manipulate the raw data to prepare it for analysis as well?
We have added language about archiving associated code.
9. Licensing
You might mention established recommendations such as http://pantonprinciples.org/
Great suggestion. Done.
It might also be worth calling attention to the fact that there is a substantial question of whether your data can indeed be protected by copyright at all and protected by certain copyright licenses. (e.g. while it may be tempting to a researcher to apply a cc-by-nc license to their data, that license is intended for "creative works" and may not cover what may instead be a collection of facts. Some further references to this issue might also help).
This level of information was actually present in an early draft and we decided that it made things unnecessarily confusing for the target audience, especially since the dividing line between copyrightable data collections and uncopyrightable data is still quite gray.v
Conclusion
Very good, glad that it returns to the theme of personal benefits to following these recommendations. (saving time, facilitating collaboration, new possibilities for research), but I think you could say more on that theme. How about "looks good on NSF data management proposal" or any other grant application, or increased citation advantage (Piwowar's work).
Good idea. Done.
Maybe add a few concrete suggestions on where to start (perhaps creating a data standard for your lab)
We couldn't find an easy way to do this that would apply broadly. However, we did add some language describing how the easiest point at which to start implementing these ideas is the planning phase prior to data collection.