2
$\begingroup$

I am fairly new to deep learning. I want to implement a deep learning paper from scratch with proper data preprocessing, model, losses etc., using an object-oriented approach in python. I want to do something similar to the official implementations I see on GitHub and other websites.

I have no idea where to start for this task. I will appreciate some guidance regarding this.

The framework I want to use is Pytorch (Since it is more pythonic way of doing things)

Improve this question
$\endgroup$
2
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Anshuman Kumar
    Mar 28 at 7:17
  • 1
    $\begingroup$ I learned about AI by finding the GPT-2 paper (cause I really wanted to understand GPT-2) and then just following citations... and more citations... and more citations... going back in time until I found something I did understand, then following the chain in reverse from there. $\endgroup$
    – user253751
    Mar 28 at 14:54

1 Answer 1

Reset to default
1
$\begingroup$

Welcome to the AI StackExchange!

If you have never implemented a deep learning paper from scratch, it might be difficult to begin with a specific paper of your choice as there will not be many other implementations to help you out or other people who know the paper in-depth.

I would advise you to start by implementing a very popular/widely cited deep learning paper. If you get stuck along the way, you can google other people's implementations to get some guidance or ask additional questions on the Data Science Stack Exchange.

Now, to give some guidance on where to start, I'll give you my usual process of implementing papers. (Note: this is not necessarily the best or the only way, nor the way I always go about it, just the best way I could summarize my approach)):

  1. Read the paper on a surface level. Try to understand the key concepts (not the small details).
    • What is the problem they want to solve?
    • What is their approach to solving the problem?
    • What are the key deep learning concepts they are using?
    • Do you understand the general structure of the architecture/method they propose?
    • What data do they use? What is the format of this data?
    • (for neural net architectures)
      • What is the loss function?
      • Why is this loss function used?
  2. Dive deeper into the stuff you don't yet completely understand. Depending on what you don't understand yet, use the citations in the papers or make use of the internet (blog-posts, Stack Overflow, etc.). Understanding the concepts used in the paper is important!
  3. Start implementing the DataLoader. Get the data in the right format (see the question on data format above). Check the paper on batch size, shuffling and other specifics (if mentioned). If there are multiple data sources, take the easiest and simplest data. Always start simple!
  4. Start implementing the architecture components. Use a batch of data to continuously test whether your code works and keep extending onwards.
  5. Implement the training process. Loop over your data, train the model using the modules just implemented and calculate the loss and backpropagate. Don't forget to reset the gradients of the optimizer at the start.
  6. Test your code, and see if it runs.
  7. Go back to the paper to read all the specifics. The type of optimizer, the number of neural network layers, whether they use biases, all of it! Implement all these specific details.
  8. Test and debug time! Test your model on the training data and see if it converges in a proper manner. Debug anything that doesn't work!

Remember, implementing deep learning papers from scratch is not an easy thing to do! Don't get frustrated if it doesn't work immediately. I can promise you it won't. Therefore, again, make sure to start with a well-known paper. Also during implementing, start easy and work your way up. Don't just start with a super deep network, with millions of parameters. Start easy.

Good luck and have fun ;)

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ "Search the internet for stuff you don't yet completely understand" - I'll add: read the damn citations for stuff you don't understand! Usually it is explained in some earlier paper and this paper does not re-explain it because it's already explained in some earlier one where it was new. Then it cites the earlier one. $\endgroup$
    – user253751
    Mar 28 at 14:55
  • $\begingroup$ @Robin van Hoorn thank you for the insight. I think this gives me basic idea of how to start in first place. This was a really nice answer. $\endgroup$
    – programmer_04_03
    Apr 1 at 9:56
  • $\begingroup$ @ShubhamDeshpnde if the question has sufficiently answered your question, please upvote it and select it as the 'correct' answer. This helps future visitors of the website to quickly find what they are looking for. $\endgroup$
    – Robin van Hoorn
    Apr 2 at 10:03
  • $\begingroup$ @user253751 Thanks for the note, you are completely correct. I edited my answer to incorporate your suggestion. $\endgroup$
    – Robin van Hoorn
    Apr 2 at 10:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .