📘 Studying the LLaMA-2 Architecture

This repository documents my learning journey while studying the LLaMA-2 architecture.
My main sources of study were:

• Umar Jamil’s YouTube Lecture on LLaMA-2
• Umar Jamil’s Simplified PyTorch Implementation Repository
• Research papers used in the individual components of the architecture, such as RoFormer and RMSNorm
• The official LLaMA-2 Paper: https://arxiv.org/abs/2307.09288

---

📂 Repository Structure

The repository is organized into the following folders:

• Clean Codes
  Contains the implementation of the Inference Pipeline of LLaMA-2 after building its architecture from scratch.
  Most of the code is adapted from Umar Jamil’s repository, which provides an excellent, simplified, and self-explanatory implementation.
  To make the learning process more engaging and connected with the slides and papers, I have added detailed explanations and images for each component of the LLaMA architecture in the llama_2.ipynb notebook. This makes it easier for readers to understand what each component does and how it fits into the bigger picture.

• Rough Codes
  Contains scratch work, experiments, and intermediate implementations created while learning.

• Notes
  My personal notes made during the learning process. I’ve tried to explain a few components that I personally found difficult to grasp through just the lecture or papers. I hope these notes help others who face similar challenges.

• Resources
  Includes additional materials such as:

  • RoFormer Paper
  • RMSNorm Paper
  • Slides by Umar Jamil
  • The LLaMA-2 Paper
  • Umar Jamil’s Lecture

---

🙏 Acknowledgements

A huge thanks to Umar Jamil for his amazing efforts in simplifying complex architectures like LLaMA-2. His slides, code, and explanations were invaluable in this journey.

---

🎯 Purpose

This repository is strictly for educational purposes.
It is meant to help learners like me connect theoretical concepts from research papers with practical implementations, while also providing additional explanations and references for clarity.

---

💬 Open for Discussions

I am always open to discussions and deep dives into Large Language Models (LLMs).
If you find this repository useful or would like to collaborate/exchange ideas, feel free to reach out or start a discussion!

Happy Learning 🚀