Visualization for NLP and LLMs

CS-GY 9223 - Fall 2025

Claudio Silva

NYU Tandon School of Engineering

2025-11-03

NLP and Large Language Models

Today’s Agenda

1. NLP Fundamentals
   • Text processing pipeline
   • Traditional vs modern approaches
2. Visualizing Model Internals
   • RNN gradient flow
   • Hidden state analysis
3. Behavioral Analysis
   • Error discovery patterns
   • Counterfactual generation
4. LLM Visualization
   • Attention mechanisms
   • Prompt engineering tools

Natural Language Processing has undergone a revolution with the advent of large language models. Today’s lecture bridges traditional NLP visualization techniques with modern LLM analysis methods. We’ll start with fundamental text processing concepts, then explore how visualization helps us understand both classical models like RNNs and modern transformer-based architectures. The transition from traditional NLP to LLMs represents not just a scale change but a fundamental shift in how we approach language understanding. Visualization plays a crucial role in making these black-box models more interpretable.

Visualization for NLP

• NLP Basic
• Visualizing Traditional NLP Model’s Internal Structure
• Visualizing Traditional NLP Model’s Behavior
• Visualizing LLMs

NLP Basic: Text Processing Pipeline

The NLP pipeline begins with raw text and transforms it through multiple stages into a form suitable for machine learning. Tokenization splits text into units (words, subwords, or characters). Normalization handles case, punctuation, and special characters. Vectorization converts tokens to numbers - from simple one-hot encoding to sophisticated contextual embeddings. Each stage involves design decisions that significantly impact downstream performance. For example, aggressive normalization might remove important signals (like capitalization indicating proper nouns), while preserving too much variation increases vocabulary size and sparsity.

NLP Basic: Word Embeddings

Word embeddings revolutionized NLP by providing dense, continuous representations where similar words have similar vectors. The key insight is the distributional hypothesis: words appearing in similar contexts have similar meanings. Word2Vec uses shallow neural networks to predict context words (CBOW) or target words (Skip-gram). GloVe combines global matrix factorization with local context windows. FastText extends Word2Vec by using character n-grams, helping with out-of-vocabulary words. These embeddings capture surprising regularities - the famous “king - man + woman = queen” demonstrates that vector arithmetic can encode analogical reasoning.

NLP Basic: Sequence Models

Sequence modeling is central to NLP because language is inherently sequential. RNNs maintain a hidden state that gets updated at each time step, theoretically capturing all previous context. However, standard RNNs suffer from vanishing gradients, making it hard to learn long-range dependencies. LSTMs and GRUs address this with gating mechanisms that control information flow. Bidirectional RNNs process sequences in both directions, capturing both past and future context. The sequential nature of RNNs makes them slow to train and inference, motivating the development of parallelizable architectures like Transformers.

NLP Basic: Text Preprocessing

Text preprocessing is crucial but often underappreciated. Common steps include lowercasing (but this loses information like proper nouns), removing stop words (but they’re important for understanding negation and questions), stemming/lemmatization (reducing words to root forms), and handling special tokens like URLs, numbers, and emojis. Modern deep learning approaches often use minimal preprocessing, letting the model learn what’s important. However, careful preprocessing can significantly reduce vocabulary size and improve generalization, especially with limited training data. The key is understanding your task - sentiment analysis might benefit from preserving emoticons, while topic modeling might safely remove them.

NLP Basic: Feature Extraction

Feature extraction transforms text into numerical features for machine learning. Traditional approaches include bag-of-words (word counts), TF-IDF (term frequency-inverse document frequency), and n-grams (sequences of n words). These create sparse, high-dimensional representations. Modern approaches learn dense features: word embeddings, sentence encoders, and contextual representations from transformers. The choice depends on your task and data. Sparse features work well for many classification tasks with limited data, while dense representations excel at capturing semantic similarity and transfer learning. Visualization plays a key role in understanding what features your model learns and relies upon.

NLP Basic: Language Model Architectures

Language model architectures have evolved from simple n-gram models to sophisticated neural architectures. N-gram models estimate probabilities based on local context but suffer from data sparsity. RNN language models maintain hidden states to capture longer context but process sequentially. Transformer-based models like GPT and BERT use self-attention to model global dependencies efficiently. The key innovation is attention mechanisms that allow models to focus on relevant parts of the input. Pre-trained language models have become the foundation of modern NLP, providing rich representations that transfer across tasks. Understanding these architectures helps in choosing the right model and visualization technique for your application.

NLP Tasks Taxonomy

NLP tasks span from low-level syntactic analysis to high-level semantic understanding and generation. Token-level tasks like POS tagging and NER identify properties of individual words. Sequence-level tasks like machine translation and summarization transform entire texts. Classification tasks like sentiment analysis and topic modeling categorize documents. Generation tasks create new text. This taxonomy matters for visualization because different tasks benefit from different visual representations. Token-level tasks use attention heatmaps and dependency graphs. Document-level tasks use embedding projections and clustering visualizations. Understanding task requirements helps select appropriate models and interpretability methods.

Token & Syntax Level

• Named entity recognition
• Parts-of-speech tagging
• Dependency parsing
• Grammatical error correction
• Word sense disambiguation
• Coreference resolution

Document & Semantic Level

• Text summarization
• Question answering
• Machine translation
• Sentiment analysis
• Topic modeling
• Dialogue systems

Visualization for NLP

• NLP Basic
• Visualizing Traditional NLP Model’s Internal Structure
• Visualizing Traditional NLP Model’s Behavior
• Visualizing LLMs

Visualization for NLP

• NLP Basic
• Visualizing Traditional NLP Model’s Internal Structure
• Visualizing Traditional NLP Model’s Behavior
• Visualizing LLMs

RNNbow: Visualizing Gradient Flow

Key Innovation

• Visualizes backpropagation gradients in RNNs
• Reveals vanishing/exploding gradient problems
• Tracks learning dynamics across epochs

Applications

• Model debugging
• Architecture comparison
• Training optimization

RNNbow addresses a fundamental challenge in training recurrent networks: understanding gradient flow through time. The visualization reveals how gradients propagate backwards through the network, exposing the vanishing gradient problem that plagued early RNNs. This tool was instrumental in motivating architectural innovations like LSTMs and GRUs. The ability to compare gradients across different training epochs provides insights into learning dynamics that are impossible to obtain from scalar metrics alone.

RNN Architecture Visualization

This visualization shows how RNNs process sequences by maintaining and updating hidden states over time. Each node represents a hidden state vector, and edges show the flow of information. The recurrent connections (horizontal arrows) carry information from previous time steps, while vertical connections show input/output flow. Color coding often represents activation magnitudes or gradient values. This unrolled view makes clear why RNNs struggle with long sequences - information must pass through many nonlinear transformations, leading to signal degradation. Understanding this architecture is crucial for interpreting gradient flow visualizations and diagnosing training problems.

Gradient Flow Analysis

Gradient flow visualization is essential for understanding RNN training dynamics. This diagram traces how gradients propagate backward through time during backpropagation. Line thickness or color intensity represents gradient magnitude. The vanishing gradient problem becomes visually apparent - gradients exponentially decay as they flow backward, making it impossible to learn long-range dependencies. This visualization technique has been instrumental in motivating architectural improvements. LSTMs and GRUs include “gradient highways” - paths where gradients can flow with minimal attenuation. Gradient clipping and careful initialization also help, but the fundamental sequential bottleneck remains.

Training Dynamics: Epoch Comparison

Gradient Evolution

• Early epochs: chaotic gradients
• Middle epochs: pattern emergence
• Late epochs: stable flow
• Convergence indicators

Comparing gradient flow across training epochs reveals how RNNs learn. Early in training, gradients are chaotic and unstable, reflecting the random initial weights. As training progresses, gradient patterns stabilize and become more structured. Successful training shows gradients maintaining reasonable magnitudes throughout the sequence. Failed training often shows gradients vanishing (approaching zero) or exploding (growing exponentially). This visualization helps identify when to stop training, adjust learning rates, or modify architectures. It also reveals which parts of the sequence contribute most to learning - often later time steps dominate due to shorter gradient paths.

Visualizing Internal Structure: RNN

Comparing gradients at different epochs of training:

Visualizing Internal Structure: RNN

Exploring vanishing gradient

Visualizing Internal Structure: RNN

Poorly Learning C

LSTMVis: Understanding Hidden States

LSTMVis pioneered the visual analysis of RNN hidden states, revealing how these models encode linguistic information. The tool allows users to select phrases and see which hidden units activate, discovering that different units specialize in different linguistic phenomena. This work demonstrated that RNNs learn interpretable representations despite not being explicitly trained to do so.

Visualizing Internal Structure: Hidden States

Hidden values in LSTM/RNN

Visualizing Internal Structure: Hidden States

Hidden States Sequence of high-dimensional vectors. What are some options for visualization?

Visualizing Internal Structure: Hidden States

Hidden States Sequence of high-dimensional vectors. What are some options for visualization?

Visualizing Internal Structure: Hidden States

Hidden States Sequence of high-dimensional vectors. What are some options for visualization?

Visualizing Internal Structure: Hidden States

User selects sequences. Configurable threshold: all hidden states in the selected sequence must exceed threshold

Visualizing Internal Structure: Hidden States

User selects sequences. Configurable threshold: all hidden states in the selected sequence must exceed threshold

Collection-Level Hidden State Analysis

Visualization Challenges

• High dimensionality
• Temporal dynamics
• Multiple sequences
• Pattern discovery

Visualizing hidden states across an entire dataset presents unique challenges. We need to understand patterns across three dimensions: hidden units, time steps, and different input sequences. Dimensionality reduction techniques like PCA or t-SNE can reveal clusters of similar hidden states, suggesting that the model groups similar inputs. Trajectory visualizations show how hidden states evolve over time for different input types. Statistical aggregations reveal which units are most active and when. This collection-level view helps identify model capacity issues (are all units being used?), specialization patterns (do units have consistent roles?), and dataset biases (do certain inputs always produce similar hidden states?).

Visualization for NLP

• NLP Basic
• Visualizing Traditional NLP Model’s Internal Structure
• Visualizing Traditional NLP Model’s Behavior
• Visualizing LLMs

iSea: Semantic Error Analysis

Overall Accuracy: 80% - But where are the failures?

A model with 80% accuracy might seem good, but understanding the 20% of failures is crucial for improvement. iSea moves beyond aggregate metrics to identify systematic error patterns. Rather than looking at random failures, it discovers semantically coherent subpopulations where the model consistently fails. This approach has revealed biases, dataset artifacts, and fundamental model limitations that would be invisible in traditional evaluation.

Visualizing Behavior: iSea

Where does the model make mistakes?

Why does the model make these mistakes?

How can we improve the model performance?

…

Visualizing Behavior: iSea

Subpopulation-Level Error Analysis is Common for NLP Models

Visualizing Behavior: iSea

Subpopulation-Level Error Analysis is Common for NLP Models

Not able to capture the errors grounded in specific semantic concepts.

Requires prior knowledge to construct subpopulations.

Visualizing Behavior: iSea

Visualizing Behavior: iSea

Visualizing Behavior: iSea

Features to Describe A Subpopulation

• Token
  • e.g., all the documents that contain “delicious”.
• Concept
  • e.g., all the documents that contain “delicious”/“tasty”/”yummy”/…
• High-level Features
  • e.g., all the documents that contain a high percentage of adjectives.

Visualizing Behavior: iSea

To describe error-prone subpopulations, we use a set of if-then rules.

Visualizing Behavior: iSea

Through iterative design process, we identified four principles of presenting the error rules:

• Principle 1: Limit the number of conditions.
  • To keep the rule interpretable.
• Principle 2: Test significance.
  • To ensure the high error rate in the subpopulation does not occur by chance
• Principle 3: Limit the cardinality of features.
  • Use low/medium/high instead of actual values (e.g., >20, <30) to keep it interpretable.
• Principle 4: Avoid negation for tokens.
  • To ensure actionable insights.

Visualizing Behavior: iSea

Automatic Error Discovery

Visualizing Behavior: iSea

Automatic Error Discovery

Visualizing Behavior: iSea

Automatic Error Discovery

Visualizing Behavior: iSea

Views to Support Learning

Visualizing Behavior: iSea

Validating

Visualizing Behavior: iSea

Validating

Visualizing Behavior: iSea

Validating

Visualizing Behavior: iSea

Interpret Errors Causes

Visualizing Behavior: iSea

Hypothesis Testing

Visualizing Behavior: iSea

Rule Editing & Concept Construction

Visualizing Behavior: iSea

Polyjuice: Counterfactual Generation

Key Features

• Generates diverse counterfactuals
• Controls perturbation types
• Reveals model robustness issues

Applications

• Data augmentation
• Robustness testing
• Bias detection

Polyjuice addresses a critical challenge in NLP model evaluation: understanding robustness to input variations. By generating controlled counterfactuals, it reveals whether models rely on spurious correlations or genuine understanding. This tool has been instrumental in discovering dataset biases and improving model robustness through targeted data augmentation.

Visualizing Behavior: Polyjuice

Visualization for NLP

• NLP Basic
• Visualizing Traditional NLP Model’s Internal Structure
• Visualizing Traditional NLP Model’s Behavior
• Visualizing LLMs

Attention Visualization in Transformers

Attention mechanisms are the core of modern LLMs, but understanding what models “attend to” is challenging. This visualization reveals how different attention heads specialize in different linguistic phenomena: some track syntax, others capture semantic relationships, and some focus on positional patterns. The multi-scale approach allows exploration from individual token relationships to aggregate patterns across layers.

LLM Architecture: Self-Attention

Self-attention is the core mechanism that enables transformers to model long-range dependencies efficiently. Unlike RNNs that process sequentially, attention computes relationships between all pairs of positions simultaneously. The attention matrix shows which tokens the model “looks at” when processing each token. Multiple attention heads learn different relationship types - some track syntax (subject-verb agreement), others capture semantic relationships (coreference), and some focus on position (attending to adjacent tokens). Visualizing attention patterns helps understand model decisions and identify potential biases or failure modes. However, attention doesn’t always equal explanation - high attention doesn’t necessarily mean high importance for the final prediction.

Multi-Head Attention Patterns

Transformers use multiple attention heads that learn to focus on different types of information. This visualization shows how different heads specialize: some become “positional” heads attending to nearby tokens, others are “syntactic” heads tracking grammatical dependencies, and some are “semantic” heads linking related concepts. The multi-head design provides redundancy and allows the model to simultaneously attend to information at different positions and representation subspaces. Analyzing head specialization helps understand model capabilities and can guide model pruning - some heads can be removed with minimal performance impact. This visualization technique has revealed that many heads in large models are redundant or inactive.

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

Visualizing LLMs

POEM: Prompt Engineering for Multimodal LLMs

POEM represents the frontier of LLM interaction design, addressing the challenge of prompt engineering for multimodal models. As LLMs become capable of processing images, video, and text together, understanding how to craft effective prompts becomes increasingly complex. POEM provides visual feedback on prompt effectiveness, suggests improvements, and helps users understand the model’s reasoning process across modalities. This is particularly important as prompt engineering has become a critical skill for effectively using LLMs.

Visualizing LLMs

Multi-Modal Interaction

Visualizing LLMs

POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models