Highlight

Conformer: Convolution-augmented Transformer for Speech Recognition

Proposes Conformer, a convolution-augmented Transformer that models local and global audio dependencies, achieving state-of-the-art speech recognition.

Transformer and CNN models have advanced speech recognition beyond RNNs; Transformers capture global interactions, CNNs local features. The authors combine both in a parameter-efficient way, proposing Conformer, a convolution-augmented Transformer that models local and global audio dependencies. It significantly outperforms prior Transformer and CNN models, reaching state-of-the-art accuracy. On LibriSpeech it reaches word error rates of 2.1%/4.3% without a language model and 1.9%/3.9% with one on test/test-other, and a 10M-parameter version reaches a competitive 2.7%/6.3%.

Based on: Conformer: Convolution-augmented Transformer for Speech Recognition · Interspeech

Curated by Aramai Editorial

Read summary →
Highlight

Wide & Deep Learning for Recommender Systems

Introduces Wide & Deep learning, jointly training wide linear models and deep neural networks to combine memorization and generalization for recommenders.

Wide cross-product features give effective, interpretable memorization but need heavy feature engineering, while deep networks with embeddings generalize to unseen feature combinations with less engineering, yet can over-generalize when interactions are sparse. Wide & Deep learning jointly trains a wide linear model and a deep network to combine memorization and generalization. Productionized on Google Play (1B+ active users), online experiments showed significantly more app acquisitions than wide-only and deep-only models; the code was open-sourced in TensorFlow.

Based on: Wide & Deep Learning for Recommender Systems · DLRS@RecSys

Curated by Aramai Editorial

Read summary →
Highlight

Universal Language Model Fine-tuning for Text Classification

Proposes ULMFiT, a universal inductive transfer-learning method that fine-tunes a pretrained language model for any NLP text-classification task.

Inductive transfer learning has transformed computer vision, but NLP approaches still required task-specific architectures and training from scratch. The authors propose Universal Language Model Fine-tuning (ULMFiT), a transfer-learning method for any NLP task, plus key techniques for fine-tuning a language model. ULMFiT beats the state of the art on six text-classification tasks, cutting error by 18 to 24 percent on most datasets. With just 100 labeled examples it matches training from scratch on 100x more data; the pretrained models and code are open-sourced.

Based on: Universal Language Model Fine-tuning for Text Classification · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning

Introduces Scaled Conjugate Gradient (SCG), a supervised neural-network training algorithm with superlinear convergence, O(N) memory, and no line search.

The paper introduces Scaled Conjugate Gradient (SCG), a supervised learning algorithm with superlinear convergence based on conjugate gradient optimization methods. It uses second-order information yet needs only O(N) memory (N = number of weights), and is fully automated with no user-set parameters and no line search. Benchmarked against backpropagation (BP), conjugate gradient backprop, and memoryless BFGS, SCG gives at least an order-of-magnitude speed-up over BP, growing with stricter error demands. It also handles sharp-curvature ravines where BP is inefficient.

Based on: A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning · Semantic Scholar

Curated by Aramai Editorial

Read summary →
Highlight

Lost in the Middle: How Language Models Use Long Contexts

Analyzes how language models use long input contexts, showing accuracy drops sharply when relevant information sits in the middle rather than the ends.

Although recent language models accept long contexts, little is known about how effectively they use them. The authors evaluate models on two tasks: multi-document question answering and key-value retrieval. Performance degrades substantially as the position of the relevant information changes, showing that models do not robustly exploit long inputs. Accuracy peaks when the needed information is at the beginning or end and drops in the middle, even for long-context models. The study offers new evaluation protocols and a clearer picture of context usage.

Based on: Lost in the Middle: How Language Models Use Long Contexts · Transactions of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Reflexion: language agents with verbal reinforcement learning

Introduces Reflexion, a framework that improves LLM agents through verbal self-reflection stored in episodic memory instead of updating model weights.

LLMs increasingly act as goal-driven agents, but trial-and-error learning is hard since traditional RL needs many samples and costly fine-tuning. Reflexion reinforces language agents not by updating weights but through linguistic feedback: agents verbally reflect on task feedback and store these reflections in an episodic memory buffer to guide later attempts. It handles diverse feedback types and sources and beats a baseline on decision-making, coding, and reasoning. Notably, Reflexion reaches 91% pass@1 on HumanEval, exceeding prior-best GPT-4 at 80%.

Based on: Reflexion: language agents with verbal reinforcement learning · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Describes SentencePiece, a language-independent subword tokenizer and detokenizer that trains directly from raw sentences for neural text processing.

SentencePiece is a language-independent subword tokenizer and detokenizer built for neural text processing, including neural machine translation. Unlike existing tools that assume pre-tokenized word sequences as input, it can train subword models directly from raw sentences, enabling a fully end-to-end, language-independent pipeline. In an English-Japanese NMT experiment, training directly from raw sentences matches the accuracy of direct subword training. Open-source C++ and Python implementations are released under the Apache 2 license.

Based on: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Conference on Empirical Methods in Natural Language Processing

Curated by Aramai Editorial

Read summary →
Highlight

Graph Convolutional Neural Networks for Web-Scale Recommender Systems

Presents a scalable graph convolutional network combining random walks and convolutions to embed items for web-scale recommendation at Pinterest.

Graph neural networks excel on recommender benchmarks, but scaling them to billions of items remains unsolved. The authors present a data-efficient GCN deployed at Pinterest that blends random walks with graph convolutions to embed items using both graph structure and node features. Novel random-walk convolutions, a harder-example training strategy, and MapReduce inference enable graphs four orders of magnitude larger than typical GCNs. On a 3-billion-node, 17-billion-edge graph, A/B tests and user studies show higher-quality recommendations than comparable systems.

Based on: Graph Convolutional Neural Networks for Web-Scale Recommender Systems · Knowledge Discovery and Data Mining

Curated by Aramai Editorial

Read summary →
Highlight

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Proposes a machine-learning method for sentiment polarity that classifies only a document's subjective portions, extracted via minimum graph cuts.

Sentiment analysis aims to identify the viewpoint underlying a text span, such as labeling a movie review as thumbs up or thumbs down. The authors propose a machine-learning method that applies text-categorization techniques to only the subjective portions of a document. Those portions are extracted using efficient minimum-cut algorithms on graphs, which makes it easy to incorporate cross-sentence contextual constraints.

Based on: A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts · Annual Meeting of the Association for Computational Linguistics

Curated by Aramai Editorial

Read summary →
Highlight

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Introduces the Waymo Open Dataset, a large-scale, diverse autonomous-driving benchmark with synchronized LiDAR and camera data plus 2D/3D annotations.

Real-world autonomous-driving data is costly, and existing datasets are limited in scale and environmental variation. The authors present a large, diverse dataset of 1150 scenes, each 20 seconds long, with synchronized, calibrated LiDAR and camera data across urban and suburban areas. By their metric it is 15x more diverse than the largest prior camera+LiDAR set, and is exhaustively labeled with 2D and 3D bounding boxes with consistent cross-frame IDs. They provide 2D/3D detection and tracking baselines and study how dataset size and geography affect 3D detection.

Based on: Scalability in Perception for Autonomous Driving: Waymo Open Dataset · Computer Vision and Pattern Recognition

Curated by Aramai Editorial

Read summary →
Highlight

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Proposes Binary-Weight-Networks and XNOR-Networks that binarize CNNs for large memory savings and faster convolutions on ImageNet.

This work proposes two efficient approximations to standard CNNs. In Binary-Weight-Networks, filters are approximated with binary values, yielding 32x memory savings. In XNOR-Networks, both filters and convolutional inputs are binary, so convolutions use primarily binary operations, giving about 58x faster convolution and enabling real-time inference on CPUs. On ImageNet, a Binary-Weight-Network version of AlexNet matches full-precision AlexNet accuracy, and the method outperforms prior binarization methods BinaryConnect and BinaryNet by over 16% in top-1 accuracy.

Based on: XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks · European Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Proposes HuBERT, a self-supervised speech representation model using masked prediction of offline-clustered hidden units.

Self-supervised speech learning faces three challenges: multiple sound units per utterance, no lexicon during pre-training, and variable-length units without explicit segmentation. HuBERT (Hidden-Unit BERT) uses offline clustering to provide aligned target labels for a BERT-like prediction loss applied only over masked regions, forcing a combined acoustic and language model over continuous inputs. Relying on clustering consistency rather than label quality, HuBERT matches or beats wav2vec 2.0 on LibriSpeech and Libri-light, with up to 19% relative WER reduction at 1B parameters.

Based on: HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units · IEEE/ACM Transactions on Audio Speech and Language Processing

Curated by Aramai Editorial

Read summary →