TL;DR: I don’t know what is the exact solution to this problem, but I do know what is not. I believe it is not possible to solve the problem generally (for all research papers at once), but it is possible to solve it partially (based on different research subfields).
Disclaimer: All platforms discussed in this blog post are very good initiatives. This post does not intend to diminish their value. It just tries to explain what kind of problems they are and aren’t solving.
Acknowledgements: I would like to thank Umut Köksaldı for his feedback on a preliminary version of this post.
The Problem
The problem statement is really simple. For almost everything on the Internet, we are able to search and locate what has been discussed for that thing so far. Except research papers. Although there are several questions and explainers here and there, it is not very clear what to do when you have a question about a research paper or you feel stuck while reading it. I wrote a better description a while ago if you are interested.
When I introduce this problem to someone, there are several methodologies/platforms that people immediately suggest by saying “Why don’t you do/use XYZ?”. In this post, I will try to go over them and explain what these platforms are trying to accomplish (spoilers: most of them do not actually attempt to solve the cold start problem) and why they can not be used to solve the cold start problem.
1) Why don’t you ask the authors?
This similar to calling the manufacturing company of your phone when you can’t figure how to turn the airplane mode off. It is a valid approach, but dramatically inconvenient. The proper way to do this is to ask it online, or more importantly, someone already asked it and some other person already gave the answer. You might be saying “hold on, a question about a research paper will likely be much more delicate than a question about a consumer phone. It may require the expertise of the author.” You are right, however, this doesn’t justify the methodology. An author can still reply publicly and they wouldn’t have to reply again and again in private. Moreover we actually wouldn’t need the expertise of the author for many questions or discussions going around. The Internet is a cumulative brain and you can not compete with it, even for answering the questions related to the research papers you yourself wrote.
2) Why don’t you ask the question in public, or by using Stackexchange?
Public doesn’t imply “conveniently locatable”. In some communities, authors are very active on Twitter, you can ask questions and start discussions publicly and they will be more than happy to answer the questions and engage in discussions. It is also possible to ask a question on Stackexchange and get answers fairly quickly. The problem with both these approaches is indexing. None of these questions are indexed by the paper. It is actually very hard to come across one of these questions when you search the title of the paper online. Don’t get me wrong, Stackexchange is one of the best things that ever happened to the Internet, it is just that we need questions and discussions indexed by research papers. Stackexchange does not provide this.
3) Why don’t you use OpenReview?
In the name ‘OpenReview’, it is a platform for publishing open reviews for papers. Reviews are mainly for authors, they provide suggestions for increasing the overall quality of the paper. The cold start problem is mainly about reading a published work, the solution will be mostly beneficial for readers. OpenReview has nothing to do with this problem whatsoever. It’s there to solve a different problem (peer review), I do have opinions about whether it is a good platform to solve problems in the peer reviewing process, but that is a story for another day.
4) Why don’t you use SciRate?
Now you are talking. SciRate’s contribution is two-fold. First, they have a rating mechanism for arXiv papers. Second, they attach a comment section to arXiv papers. However, there are two important problems with their approach:
- Too general: SciRate covers all arXiv papers. This means that all communities for different research subfields need to coexist on the same website. Although this looks quite normal (we have Reddit afterall) managing different communities on the same platform is a very hard task. If someone is willing to tackle this problem for all subfields at once, the platform should have a fine grained subcommunity mechanism, just like Reddit. They have to be moderated and structured differently. Note that by a subcommunity; I mean “cryptography”, “statistical physics”, “number theory”. Not computer science, physics and math. Two different subfields (even under the same subject) have very different communities. Some examples for how they differ are as follows:
- Number of people
- Number of research papers published every year
- Type of venues that they publish (conference oriented/journal oriented)
- Type of discussion they need to have
- General jargon being used between researchers
- Whether they have open access preprints
Given these circumstances, if a website contains papers from different subfields, it is doomed to get taken over by a single subfield without a lot of restructuring and moderation. For SciRate, this subfield is “quantum physics”. You will see what I mean if you browse SciRate for ten minutes. A better aproach could be having different platforms tailored towards different subfields.
- A comment section is not enough: Attaching a comment section to arXiv papers is a very good move for solving the cold start problem. However, a comment section doesn’t have enough structure to maintain questions and discussions in the long term. A single question is worth its own thread. One important benefit we have with threads is that once we have a thread, it’s possible to accept answers just like we do in Stackexchange. Moreover, it is more likely to be indexed properly by search engines compared to a comment. Since the thread will have a dedicated url on the platform.
There can be other problems with how SciRate operates. But these two were the ones that I find most related to the cold start problem.
5) Why don’t you use PeerJ?
PeerJ is using the same approach of adding comment sections to papers. However, instead of arXiv, they have their own publishing platform, the comment section is basically an additional selling point to use their platform. They did a very decent job for implementing how comments work. It is possible to create Stackexchange like Q&A style comments and more general discussion comments. Moreover, since the papers published on their platform are HTML5, it is possible to ask questions that show up near the margins of the paper.
Margin referencing is a cool feature, Fermat’s Library beatifully implements this for their paper explanation platform (by jotting down public notes to the margins). However, I am not sure whether it would be easy to organize what’s being written in the presence of many users.
The main problem I have with PeerJ is that it is a whole new publishing platform. I think the use of HTML5 papers increases the potential to be more interactive than PDFs. However, we have a lot of papers that has been published so far. The ideal platform should accumulate public knowledge for older papers as well. It is true that we need a paradigm shift for publishing, but the cold start problem will remain for older papers. Nevertheless, I appreciate the effort that PeerJ put for attempting to create a more modern publishing experience.
There are many other platforms that I haven’t listed here, the ones discussed here are just different representatives for different kind of platforms that I wanted to go over.
Your community might not be ready to tackle this problem.
Although I have mentioned that it is better to solve this problem seperately for communities of different subfields. Not every community has the capacity to approach this problem. There are several properties that a subfield should have in order to start thinking about this problem:
- Open Access culture: If you can not access the papers that are published in your field without a paywall, there is no point on discussing this further. Solve that problem first.
- Number of preprint servers & Number of papers published: If you have too many papers in your subfield, there are two problems. First, people are not reading most of the papers. Second, it is very hard to organize the papers published in your field. The second might be the cause of the first. For example, there were 9122 papers submitted to NeurIPS 2021 and 26% was accepted to the main conference [1]. That’s roughly 2000 papers for a single conference. That’s not right and should be fixed. I am not going to go into the details of fixing this problem, but the ML community is unlikely to successfully approach the cold start problem without tackling this problem first.
- Active online community: Any platform that aims to accumulate knowledge around published papers, the platform needs to be used by many people in order to function properly. If we are creating a platform for a specific subfield, that field should have an active online community to make the website work.
An example community: The academic community of cryptographers has a very good potential of solving the cold start problem. I will go over the same points above to explain why:
- Almost all cryptography papers are published under the same preprint server. The IACR ePrint Cryptology Archive. All preprints are freely accessible in a convenient way. Moreover, they are identified with a fairly simple methodology (remember, we need to index discussions by papers). Every paper is identified with it’s publication year followed by a sequence number within that year. For instance, the first paper published on the preprint server in 1996 has id 1996/001. It is accesible by eprint.iacr.org/1996/001.
- Every year, there are around 1500 papers submitted to ePrint. This includes several flagship conferences in cryptography. Hence, the community doesn’t have excessive amounts of papers.
- The community has a very active online presence. There are forums and mailing lists for specific topics that require lengthy discussions (see the public forum for Post-Quantum Cryptography Standardization Process ). Many of the academics are also very active on Twitter and it is possible to see several cryptographers discussing a specific topic almost everyday.
As you can see, the cryptography community sets a good example for the bare minimum requirements for tackling this problem. That’s why I am also working on a platform to fix the cold start problem, specifically for the cryptography community. Here are some basic requirements (in order of importance) that I have set before starting to develop the platform:
- Single page per paper: There should be a single page that lists all question and discussion threads created related to that paper so far. This was the top priority because it is necessary for indexing the discussions by papers.
- Main Resource Topics: Each paper should have a main resource thread. This thread will include the basic metadata: title, authors, abstract and a link to the paper on the preprint server. Underneath this thread, people can post resources related to the paper. This includes implementations, conference talks, explainers etc. People can also post previous discussions that happened somewhere on the Internet. This specific thread will basically crowdsource all additional material about a paper.
- Separation of questions and discussions: Questions and discussions should have different formats and there should be an indicator to signify whether a thread is a question or a discussion. Moreover, unlike discussions, question threads can be marked as solved.
- Spanning multiple papers: A question or a discussion thread should be able to span more than one paper. In this case, the thread should appear in the pages for both papers.
- Moderation is an inside job: The community moderators should be from the community that the website is targeting to solve the cold start problem for. Not arbitrary site owners.
- Tracking Author Users: There should be a verification process for authenticating authors that has a paper on the preprint website. After this process is complete, all threads mentioning a paper should display the respective “Author users”.
- Search: It should be possible to search through all threads that is present on the website.
Answers to opposers
There are two main questions raised by people who find the idea of public discussions harmful. I also want to address them here shortly:
What if an author doesn’t want their papers to be discussed publicly?
First of all, I don’t see a single reason why. But now that you ask, the answer is simple. There is no mechanism to protect any kind of public discussion about your paper anyways. As we have seen before, these discussions already happen in different websites. Making them easier to find is a good thing.
What if someone comes up with a decent idea during a discussion? ?Who gets the credit?
Well, that’s their idea and if they actually sit and write a paper out of it, they obviously get the credit. You can also offer to collaborate with them. If you are seriously afraid of this, I think there is a fundemental problem with how you approach science.
If you are really that competitive about your work, I would recommend to take the following attitude from George Hotz related to this question: [2]
I am going to tell you what I am doing, and you can try to compete. But I’ll still crush you.
Verdict
I don’t think it is possible to solve this problem for all subfields at once. That requires a platform that is as big as Reddit. Funding gets in the way of such big projects and it gets more complicated. We need non-commercial platforms tailored towards specific subfields to satisfy their needs. I am quite confident that there are many subfields that can solve this problem with some community effort. Moreover, some subfields should improve their community mechanisms to at least qualify for attempting to solve this problem. Lastly, no matter what the platform is doing, it will only be useful if people use it.
[1] https://www.vinai.io/an-overview-of-neurips-2021s-publications/