Canvas & Ratio
Choose your destination platform format
Layout Template
Choose a content structure for your slides
Preset Themes
Typography & Sizing
Brand Kit Customization
AGENCYConfigure brand assets for headers & footers
Outro Slide CTA
Customize your closing call-to-action slide
Background Pattern
Build Your Carousel
Drag and drop any post card below onto a slide, or use the quick buttons to insert content/images instantly!
How LLMs understand relative positions of input words, clearly explained:
RoPE (Rotary Positional Embeddings) revolutionised the way positional information is encoded in LLMs and it's widely used by models like Llama-3. Today, I'll clearly explain what they are & how positional embeddings evolved over time. Let's go! 🚀
Why Positional embeddings❓ Let's first understand the concept of positional embeddings and why they are essential in the first place. Without positional embeddings a transformer doesn't understand the relative arrangement of words in a sentence.
Absolute Positional embeddings So, let's start with the simplest way to encode position of tokens (words) in an input.
Problem with absolute positional embeddings! ⚠️ Each positional embedding is independent of each other, they don't capture relative arrangements/positioning. Check this out👇
Next up, let's talk about relative positional embeddings! They definitely overcome the limitations of absolute positional embeddings, but they come with the drawback of increased parameters. For a sequence of length N, we require 2N+1 positional embeddings.
Introducing Rotary Positional Embeddings (RoPE), the best of both worlds! Instead of adding a token & its positional embedding, we rotate the token embedding by a fixed factor (theta) depending on its position in the sequence. Imagine our token embeddings are 2-dimensional!
RoPE preserves relative position and relation. For example, in the scenario below, the embedding vectors for the words 'love' and 'Tennis' will have the same cosine similarity as long as their relative positions stay the same, regardless of the sequence length. Check this out👇
Mathematical representation. This can be easily generalised to N dimensions by taking a pair at a time. (I have also shared my detailed blog on self-attention at the end) Check this out👇
If you interested in: - Python 🐍 - ML/AI Engineering ⚙️ 1. Find me → @akshay_pachaar ✔️ 2. Subscribe to our Newsletter → @DailyDoseOfDS_ and get a FREE eBook, covering 150+ core DS/ML lessons! Cheers!