vishal bakshi
welcome to my blog.
HuggingFace’s Default KV Cache and the flash_attn_varlen_func Docstring
A deep dive into understanding flash_attn_varlen_func’s docstring’s causal masks (for seqlen_q != seqlen_k) by exploring Hugging Face’s KV Cache (DynamicCache) in model.generate() with hands-on Q/K shape inspection. Unravels “bottom-right alignment” and why flash_attn_func gets called.
Initial Experiments with Imagenette
This post walks through initial 5 epoch experiments on the small Imagenette dataset as I try to get onto the leaderboard.
Understanding the Mean Shift Clustering Algorithm (and PyTorch Broadcasting)
This post walks through a Python implementation of the Mean Shift Clustering algorithm, detailing the journey from a basic concept to a GPU-optimized version, with a key focus on understanding and visualizing PyTorch broadcasting for batch processing. We explore how manipulating tensor shapes is crucial for performance and uncover insights into how 3D tensors can be intuitively understood as grouped 2D tensors.
Exploring Precision in ColBERT Indexing and Retrieval
In this blog post I show how using full precision (fp32) versus mixed precision (amp) results in different ColBERT index artifacts (centroids.pt and ivf.pid.pt), Recall@10 and retrieved passages.
The Evolution of Matrix Multiplication (fastai course Part 2 Lessons 11 and 12)
In this blog post I walk through 10 different implementations of matrix multiplication in python, Numba and PyTorch, comparing execution times on matrix multiplications between a 5-digit subset of MNIST and a single weight matrix, as well as matrix multiplications between the full 50k-image MNIST dataset and the weight matrix. My two main takeaways: 1) when in doubt, use PyTorch’s .cuda with the @ operator, and 2) different matrix multiplication algorithms scale differently!
DataInspector with BinPackCollator: Inspecting Packed Dataloader Items
I learn about LLM-Foundry’s BinPackCollator and write a custom Composer callback to inspect the data and confirm that multiple sequences are packed in each batch item, leading to using 95% loss-generating tokens instead of 5%!
Comparing RAGatouille and ColBERT Indexes and Search Results
In this blog post I answer two questions: 1. For a given document collection and indexing configuration, do RAGatouille and ColBERT produce the same index? and 2. For a given index and search configuration, do RAGatouille and ColBERT retrieve the same passages/Recall@10?
TIL: Resolving RAGatouille OOM Error and faiss-gpu Warning
A couple of fixes as I work on indexing large document collections (6M+) using RAGatouille.
Understanding Sequence Packing - Initial Musings
A hands-on investigation into how sequence packing interacts with Flash Attention in HuggingFace Transformers. Through print statements and code exploration, I discovered that position_ids are crucial for sequence packing to work correctly—without them, the wrong Flash Attention function gets called, leading to incorrect outputs and loss values. This post walks through the debugging process, comparing packed sequences with padded batches, and reveals the critical requirement for properly constructed position_ids in sequence-packed training.
Initial Manual Scoring Results for TinyStories Models
A detailed breakdown of my manual evaluation of three TinyStories language models (1M, 8M, and 28M parameters) across six capability categories. I share scoring methodology, surprising findings about emergent reasoning in small models, and comparisons to the original TinyStories paper. This analysis establishes baselines both for my own model training project and my eventual LLM Judge, and reveals how different capabilities scale with model size.
Curating Evaluation Prompts, Defining Scoring Criteria, and Designing an LLM Judge Prompt Template
Designing simple, robust, effective evals is one of the most enjoyable experiences in ML. It’s truly empowering to take something squishy like language and build structure around it to consistently track performance. In this post, I’ll walk through my process for the TinyScaleLab project, covering how I approached curating evaluation prompts, defining scoring criteria, and designing a template for an LLM Judge.
TinyScaleLab Update: Training Cost Analysis and Evaluation Infrastructure Plans
I share my progress on the TinyScaleLab project where I’m studying small language models trained on the Tiny Stories dataset. I discuss the architecture of my four model sizes (5M, 25M, 60M, and 125M parameters), compare training costs between L4 and A100 GPUs, and outline my plans for developing an evaluation framework using LLM judges. This project aims to investigate both the training dynamics of tiny models and their language capabilities across grammar, context tracking, factual knowledge, reasoning, creativity, and storytelling.
TinyScale Lab Update: Setting Eval Targets and Generating Completions for LLM Judge Development
In this TinyScale Lab update, I tackle how to evaluate language models before building them. Using the TinyStories paper, I set target scores for my upcoming 5M-125M parameter models and generate reference completions using existing TinyStories models (1M, 8M, 28M).
TinyScaleLab: Bridging Training Dynamics and Model Capabilities
In TinyScale Lab, I’m exploring the connection between training dynamics and model capabilities using tiny language models (3M-120M parameters) as research proxies. This project bridges insights from the TinyStories and Small-scale proxies papers to understand how training stability affects emergent capabilities like grammar, consistency, and reasoning. By demonstrating that meaningful ML research is possible with modest computational resources, I hope to make AI research more accessible and democratized for resource-constrained researchers worldwide.
LossInspector: A Deep Dive Into LLM-Foundry’s Next-Token Prediction with a Custom Composer Callback
I’m working on a research project where we’re fine-tuning small models with various techniques and datasets using LLM-Foundry. As part of our infrastructure setup, I wanted to thoroughly understand how a batch of data is prepared, and how the outputs of a model, along with the labels, are passed to the loss function. Enter the custom Composer callback LossInspector!
Logging Data Types for Activations, Gradients, Weights, Optimizer States and Loss during Training with LLM-Foundry
I write a custom Composer callback (with lots of Claude’s help!) to log data types of different entities during mixed precision LoRA fine-tuning. When the model is in fp32, all entities except activations are in fp32 (activations are in bf16).
Understanding Python Descriptors
While vibe coding with Claude, it introduced me to unfamiliar behavior: using the __get__(obj) method to convert a function to a bound method of the given object. This was necessary to monkey patch the self attention module’s forward pass to log input data types as register_forward_hook only works on positional arguments (which LLaMA’s self attention module doesn’t have, it only has keyword arguments). This led me to do a deep dive into understanding descriptors with the helpful Descriptor Guide in the Python docs, which I walkthrough in this blog post.
RAGatouille/ColBERT Indexing Deep Dive
In this technical deep dive, I recreate the entire RAG indexing pipeline by directly using the internal methods of RAGatouille and ColBERT libraries. I methodically build all indexing artifacts from scratch - from processing document collections and sampling embeddings to calculating centroids with k-means clustering and compressing document vectors with residual encoding. If you’re curious about what actually happens when you call RAG.index(), this video breaks down the full process step by step.
Memory Profiling raw ColBERT and RAGatouille
I use the memory-profiler library to log memory using for different indexing functions for raw ColBERT and RAGatouille indexing operations for 100k, 250k, 500k, 1M and 2M collection sizes. In general, RAGatouille uses more memory than raw ColBERT.
Estimating Storage and CPU RAM Requirements for Indexing 12.6M Documents
I index 100k, 250k, 500k, 1M and 2M documents using T4 and RTX6000Ada instances and estimate the storage and CPU RAM requirements for a 12.6M document collection.
Evaluating the DAPR ConditionalQA Dataset with RAGatouille
I calculate the Recall@10 metric for answerai-colbert-small-v1 retrieval (via RAGatouille) on the ConditionalQA dataset (via UKPLab/DAPR dataset) using the pytrec and ranx libraries.
DoRA’s Magnitude Vector
In this blog post I highlight a key difference I saw between Raschka’s and peft’s implementation of DoRA.
Recreating the PLAID ColBERTv2 Scoring Pipeline: From Research Code to RAGatouille
In this blog post, I walk through the colbert research codebase (via AnswerAI’s RAGatouille) and work my way line-by-line through the 4-stage PLAID scoring pipeline to recreate RAGatouille results for a toy example of 1 query and 3 documents.
Scoring Full Text and Semantic Search on Chunk Sizes from 100 to 2000 Tokens
In this blog post, I run retrieval on three differently preprocessed datasets using four retrieval methods from chunk sizes 100 to 2000 tokens, using my fastbook-benchmark dataset to auto-score the results. Surprisingly, full text search yields the best MRR@10 (0.67) and Recall@10 (0.95) for a chunk size of 2000 tokens.
Implementing Image-to-Image Generation for Stable Diffusion
In this blog post I successfully implement image-to-image generation in the diffusion loop provided in Lesson 10 of the fastai course.
Evaluating 4 Retrieval Methods with 6 Chunking Strategies on my fastbook-benchmark Dataset
In this blog post, I perform retrieval on the fastbook chapter documents using 24 different retrieval method-chunking strategy combinations, auto-scoring using my fastbook-benchmark dataset.
Implementing Negative Prompting for Stable Diffusion
In this blog post I successfully implement negative prompting in the diffusion loop provided in Lesson 10 of the fastai course. I also explore some other relatively unsuccessful implementations that were interesting and informative nontheless.
Training Textual Inversion Embeddings on Some Samurai Jack Drawings
In this blog post, I recap my experience (and results) with textual inversion embeddings trained on 6 sketches I created of Samurai Jack.
Comparing Cosine Similarity Between Embeddings of Semantically Similar and Dissimilar Texts with Varying Punctuation
In this blog post, I calculate the cosine similarity between different embeddings for texts that have varying types of punctuation and semantic similarity
Establishing a Semantic Search (Embedding Cosine Similarity) Baseline for My fastbookRAG Project
In this blog post, I experiment with 6 chunking/retrieval strategies to retrieve context from an array of text embeddings sufficient to answer 80.31% of the 193 fastbook end-of-chapter questions from Part 1 of the course.
Conducting a Question-by-Question Error Analysis on Semantic Search Results
In this blog post, I conduct a detailed error analysis of 29 questions (from a set of 193), where none of the 6 semantic search methods retrieved sufficient context to answer them. I examine each question, categorize the errors, and discuss potential improvements and implications for future work.
Generating a GIF Animation Using Stable Diffusion
In this blog post I repurpose the code provided in Lesson 9/10 of the fastai Part 2 course to generate an animation GIF transitioning from a picture of a skunk to a picture of a puppy.
Calculating the Ratio of Gradients in an Image
In this blog post I use the OpenCV library to calculate the ratio of the sum of non-zero x- and y-gradients to the sum of non-zero original pixels of an image. The serif font has consistently larger ratios than the sans serif font.
Calculating the Ratio of Corners in an Image
In this blog post I use the OpenCV library to calculate the ratio of the sum of non-zero corner pixels to the sum of non-zero original pixels of an image. The serif font has consistently larger ratios than the sans serif font.
Calculating the Ratio of 2D FFT Magnitude and Phase of a Text Image
In this blog post I use NumPy to calculate the ratio of the mean of 2D FFT magnitude to the count of non-zero binarized pixels (“FFT Magnitude Ratio”) and the ratio of the sum of the absolute value of 2D FFT phase to the sum of binarized pixels (“FFT Phase Ratio”). Both ratios are consistently larger for images with serif text.
Conducting a Question-by-Question Error Analysis on Full Text Search Results
In this blog post, I conduct a detailed error analysis of 39 questions from a set of 202, where none of the 6 full text search methods retrieved sufficient context to answer them. I examine each question, categorize the errors, and discuss potential improvements and implications for future work.
Establishing a Full Text Search (BM25) Baseline for My fastbookRAG Project
In this blog post, I experiment with 6 chunking/retrieval strategies to retrieve context from a sqlite database sufficient to answer 76.7% of the 202 fastbook end-of-chapter questions from Part 1 of the course.
Comparing ~100k Random Numbers Generated with Different Methods
In this blog post, I generate close to 100k random numbers using 5 different methods: ANU Quantum numbers, Python’s random module, NumPy, PyTorch and a custom implementation from Lesson 10 of the fastai course (Part 2). I am surprised by the results!
Iterating on Full Text Search Keywords using claudette
In this blog post, I use Answer.AI’s claudette library to iteratively improve keywords generated for sqlite’s full text search.
Generating Full Text Search Keywords using claudette
In this blog post, I use Answer.AI’s claudette library to interface with the Claude-3.5 Sonnet API.
Using Hybrid Search to Answer fastai the Chapter 1 Questionnaire
In this blog post I use different approaches to combine FTS5 (keyword search) and Cosine Similarity (semantic search) to retrieve context necessary to answer questions about Chapter 1 of the fastai textbook.
How Does Stable Diffusion Work?
In this blog post I review the material taught in Lesson 9 of the fastai course (Part 2: Deep Learning Foundations to Stable Diffusion).
Using Full Text Search to Answer the fastbook Chapter 1 Questionnaire
In this blog post I’ll walk through my experiments of using sqlite full text search to retrieve context relevant to answering chapter review questions. This is part of a larger fastbookRAG proejct I’m work on.
Calculating the Flesch Kincaid Reading Grade Level for the financial_phrasebank Dataset
In this blog post I calculate the Flesch Kincaid reading grade level for the financial_phrasebank dataset and find that it’s much higher than the average TinyStories reading level.
Paper Math: rsLoRA
In this blog post I think out loud as I attempt to understand pieces of the math presented in the rsLoRA paper.
Training Collaborative Filtering Models on MovieLens 100k with Different Weight Decay Values
In this notebook I explore the question—how does the wd (weight decay) parameter affect model performance and weight distributions? I use the MovieLens 100k subset as the dataset.
Improving Kaggle Private Score with Multi-Target Classification
In this notebook I apply Jeremy Howard’s approach to multi-target classification in fastai to improve a Kaggle submission score.
Paper Summary: RewardBench
A summary of research benchmarking reward models.
Recap: HMS HBAC Kaggle Competition
A recap of what and how I did on the Harvard Medical Harmful Brain Activity Classification Kaggle Competition.
Recap: My First Live Kaggle Competition
A recap of what and how I did on the Multi-Class Prediction of Obesity Risk Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 8
In this notebook I apply to my large ensemble Jeremy Howard’s approach in the “Scaling Up - Road to the Top, Part 3” notebook.
Paddy Doctor Kaggle Competition - Part 7
In this notebook I run the code from Jeremy Howard’s “Scaling Up - Road to the Top, Part 3” notebook.
Paddy Doctor Kaggle Competition - Part 6
In this notebook I work through Jeremy Howard’s Live Coding 13 video in which he finishes working on the Paddy Doctor Disease Classification Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 5
In this notebook I work through Jeremy Howard’s Live Coding 12 video in which he continues working on the Paddy Doctor Disease Classification Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 4
In this notebook I work through Jeremy Howard’s Live Coding 11 video in which he continues working on the Paddy Doctor Disease Classification Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 3
In this notebook I work through Jeremy Howard’s Live Coding 10 video in which he continues working on the Paddy Doctor Disease Classification Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 2
In this notebook I work through Jeremy Howard’s Live Coding 9 video in which he continues working on the Paddy Doctor Disease Classification Kaggle Competition.
Paddy Doctor Kaggle Competition - Part 1
In this notebook I work through Jeremy Howard’s Live Coding 8 video in which he starts working on the Paddy Doctor Disease Classification Kaggle Competition.
fast.ai Chapter 7:Test Time Augmentation
