This advice should be most relevant to people studying machine learning (ML) and natural language processing (NLP) as that is what I did in my PhD. Having said that, this advice is not just limited to PhD students. If you are an independent researcher, want to start a PhD in the future or simply want to learn, then you will find most of this advice applicable.
Pick and choose. Everyone is different. You will have the most success if you adapt the particular advice to your situation and do what works for you.
- Read broadly.
- Work on two things.
- Be ambitious.
- Be proactive.
- Write a blog.
- Keep a source of positive energy.
- Play to your strengths.
- Intern or visit a university.
- Play the long game.
1) Read broadly.
While a PhD encourages you to delve deep into a specific topic, you can add value by making connections between different topics or entirely different fields. The papers that draw such connections can often be insightful. Many ideas in deep learning take inspiration from other fields such as biology (Hinton et al., 2014), neuroscience (Wang et al., 2016), physics (Cohen et al., 2019), and many others.
In order to have a rich repertoire to draw on for inspiration, try to cultivate diverse interests. Look beyond your immediate horizon. Attend summer schools in other areas. Connect with people from other labs. Talk to people outside your subarea at conferences. Read papers from different disciplines.
ArXiv is a great source of research papers but staying up-to-date with the daily arXiv digest feels like drinking from a firehose. Instead, I use services such as arXiv sanity preserver, arXivist, my Twitter feed, and recommendations from friends to stay up-to-date and to seek out different topics. I also generally prefer to read 10 papers superficially rather than one paper in-depth (as suggested by Jeff Dean). With a search-able paper management system (I use Mendeley) you can always go back and reread the most relevant ones.
It can be helpful to dabble in different areas early in your PhD to get a sense for what interests you. Once you have found something, focus on the problems that you deeply care about. Think about the narrative you'd like to tell as part of your thesis.
2) Work on two things.
While it is good to complete a project before starting a new one, working on a single project has downsides. If the project is not going well, your motivation and well-being may suffer. If you hit a roadblock, you can do nothing but grind until you resolve it. Developing such resilience is important but may at times come at a high mental cost.
Instead, I've found it useful to work on two projects at once to keep my sanity in check. If you hit a wall on one project, you can spend some time working on the other. This allows you to free up your mind and gain a new perspective, which may help you resolve the problem. If one of the projects is going well, this may also give you a boost to make progress on the other.
To minimise context-switching, I generally try to work on one project each day. It is also helpful if both projects are in similar areas so that you can apply what you learn on one project to the other one.
3) Be ambitious.
“Shoot for the moon. Even if you miss, you'll land among the stars.”
—Norman Vincent Peale
Another benefit of working on two projects is that it allows you to be more daring. You can work on a relatively safe project and one that is high-risk but also may be more impactful. The safe work ensures that you will graduate. The high-reward work may have a larger impact.
Ambitious projects demonstrate that you are creative and can come up with new ideas. Both are extremely valuable qualities. And even if such projects fail, they may lead you to discover unexpected insights that can lead to a publication.
Ambitious, however, does not mean that you should cater to the largest possible audience. A high impact can also be concentrated in a small community. A good indication of whether something you are working on is impactful is whether you'd be excited if it was published by someone else. In the end, you want to be known as someone that challenges the status quo and charts their own course.
The PhD is often painted as a solitary affair, a lone journey on the quest towards knowledge. While you need to show substantial work that is your own in order to graduate, that does not mean that you are all on your own.
On the contrary, being able to collaborate is an important skill that you will need later on. Many projects with a large impact in ML and NLP such as AlphaGo or OpenAI Five have been developed by a team. Whether you are part of a larger team or leading a group, you will have to collaborate with others.
Collaboration dynamics are more fluid compared to the adviser-PhD relationship, which is typically well defined. Collaborations are about building trust and mutual respect. Successfully navigating collaborations takes practice. In collaborations, particularly if they are remote, it is important to communicate clearly and to set expectations.
If you are working on two projects, make one of them a collaboration. Collaborating with someone different from your advisor introduces you to a new perspective and will allow you to learn more than working on your own.
If you are based in a lab, collaborating with one of your lab mates is often the easiest choice. However, connecting and collaborating with people in other institutions may often be beneficial long-term.
5) Be proactive.
This is probably the most important piece of advice. Don't restrict yourself to the people in your immediate circle.
Reach out to people. The main value of conferences is in bringing people together. Before a conference, look up who is going (by checking authors of accepted papers) and email them. Try to be respectful, briefly introduce yourself, and state why you'd like to meet them (a useful mnemonic is Inigo Montoya's Guide to Networking Success). Most senior people make time for such meetings. Try to talk to many people and particularly seek out those who are not already well-known.
Outside of conferences, it is often useful to ask people who have worked in your area for research advice via email. It's amazing to see how many people in our field are genuinely helpful. Proper email etiquette is important, however and makes it more likely that a busy researcher will respond. In particular, you should make it clear that you've done your research and explored alternative solutions before contacting them.
Beyond advice, such connections may lead to other opportunities further down the line: job offers, collaborations, mentorship, and even friendship. Many of my collaborations started through such connections—meetings at a conference, a cold email, a Twitter message. The important thing is that they are based on mutual interests and respect. So be conscious of other's people time. In addition, early career researchers with shared interests will often be much more open to collaborations than senior researchers who already have many commitments.
Being proactive also relates to how you view and talk about your research: Make it easy for other people to discover your work by highlighting it on your website, talking about it online, and writing a blog.
6) Write a blog.
Blogging has many advantages. It allows you to practice writing—and to learn to enjoy it. In order to finish your PhD, you will have to write a thesis, which can be an excruciating process. Blogging provides the training ground that prepares you for the thesis marathon.
From a research perspective, it allows you to practice communicating and explaining things clearly. Both are qualities that differentiate the best from mediocre research papers. In fact, clear writing is important both to get your paper accepted and for high impact. In contrast to the hyper-compact format of research papers, a blog allows you to experiment and to find your own voice.
A blog can also be a great medium to present and share your work. A great blog post about a paper does not just reiterate its main findings but complements it. A blog can be much more flexible than a paper: You can highlight interesting connections, provide the reader with a broad overview of the background literature and future directions, walk through an illustrative example, highlight code snippets or qualitative examples, show interactive visualisations, or perform an in-depth error analysis.
Another great way to start blogging is to discuss what you have just become knowledgeable about. Rachel Thomas puts this as "you are best positioned to help people one step behind you". If you have just delved into a specialised area, why not save others the time and summarise the work and your insights. Most of my blog posts—from gradient descent to word embeddings—started this way. If you have just learned how to do something cool, tell others about it. Conversely, if you want to learn about a certain topic but cannot find information about it online, consider creating that resource yourself. Starting your own blog has never been easier.
Having a blog is the single thing that has led to the most positive interactions throughout my PhD. ML and NLP have become so large that even if you write about a niche area, people will be interested. While I still feel anxious when I publish something, the response has always been worth it. In general, try to ignore unconstructive feedback and remember that the community appreciates genuine and honest voices.
7) Keep a source of positive energy.
External rewards such as paper acceptances are sparse, so leveraging intrinsic rewards is often necessary.
The most natural way to stay positive and energised in research is to work on something that excites you and to follow your curiosity. Depending on your funding or position, you might not be able to choose what you work on. In those cases, try to find a particular angle that excites you. Even an application of an existing algorithm can shed light on new and unsolved questions.
A PhD can be draining at the best of times. So is important to build a support network that you can rely on. Surround yourself with positive people that support your ideas and ambitions.
At the same time, find an activity that you can fall back on to give you positive energy when things don't go as planned. This can be a collaboration, a side project, a hobby, exercise, meditation, or something else. For me, blogging filled this need. Compared to the long stretches of radio silence during peer review, writing, publishing and receiving feedback on a blog all within a couple of days feels liberating.
In the end, the most important resource is not the amount of compute you have, but your personal well-being. A crashed GPU can be rebooted; a burnt out GPU can't be fixed.
8) Play to your strengths.
“The most value comes from doing something no else can do, or no one else has thought of.”
With the increasing interest in ML and NLP, finding a fruitful undisturbed research topic can be challenging. A good strategy is to work on something that you are in the best position to tackle. Your ideal research topic sits at the intersection of work that is impactful, work that you are passionate about, and work that you are uniquely suited for.
What makes you uniquely suited can be one of many things: Your background; your knowledge of a particular technology, method, language, or data; your personal preferences. Do you come from a non-CS background? Use this as inspiration for your work. Are you a visually creative person? Supplement your blog and papers with graphs and analyses that will inspire others. Are you a strong coder? Implement technically challenging models. Are you great at maths? Prove your claims mathematically.
Another strength can be your network and the diversity of perspectives that you have access to. So locate others that complement your strengths, whether as advisors, mentors, or collaborators.
9) Intern or visit a university.
The best way to make meaningful connections is to collaborate closely with people and to get to know them in-person. Internships and research visits are both excellent opportunities to expand your network as they enable you to work side-by-side with a group of talented people day-to-day.
They also allow you to get a feeling for how research is done in another environment. If you are considering whether to go into academia or industry, seeing first-hand how research is done in industry is an invaluable data point. A research visit or internship can also help you decide whether you would enjoy joining a lab or company at a later point.
Lastly, both are amazing learning experiences as you often will need to get familiar with a new tech stack or new research area. Through the guidance of a knowledgeable mentor different from your advisor, you will also be able to focus on different aspects of your personal growth.
10) Play the long game.
Most of us are where we are because someone took a bet on us early on. My first research visit only happened because my host took a chance on me. So if you get the chance, pay it forward. Maximise not just the expected reward of yourself but of others around you.
While being at a big institution gives you access to an initial network, in the long term you want to develop a network of smart people that you can work with. One of the best ways to build a network is by being proactive and helping people as much as you can. This can be through writing blog posts or libraries, publishing tutorials and courses, doing podcasts, reimplementing models, or helping with open-source software. If you do this consistently, you will develop a reputation for being diligent and helpful and people will want to work with you.
Generally be kind to others. Assume good intentions. Be generous in giving praise and attribution. Don't hold grudges. In fact, being nice is one of the best things you can do to be successful (see Paul Graham's Mean People Fail). Being nice also has a recurring benefit as conferences are effectively—beyond the presentation and exchange of ideas—a yearly reunion of the friends you make along the way.
Take care of yourself. Work hard but get enough sleep and exercise. Take the time to learn new things. Work on things that you are not an expert at. In the end, always remind yourself that while a PhD is supposed to culminate in a thesis, the more important outcome of a PhD is a better version of yourself.
References and inspiration
Finally, here are a few more pieces that served as inspiration:
- Andrey Karpathy's A Survival Guide to a PhD
- The Frontiers in Natural Language Processing Expert Responses at the Deep Learning Indaba 2018
- John Schulman's An Opinionated Guide to ML Research
- Andrey Kurenkov's Lessons Learned the Hard Way in Grad School (so far)
- Volkan Cirik's PhD 101
- Tim Dettmer's How to Pick Your Grad School
- Isabelle Augenstein's Increasing Well-Being in Academia
- Richard Hamming's You and Your Research
- Fei-Fei Li's De-Mystifying Good Research and Good Papers
- Sam Altman's How To Be Successful and associated Twitter thread
- Stuart K. Card's The PhD Thesis Deconstructed
- Tweets from Himan Abdollahpouri, Chip Huyen, and many others