Highlight

Graph Neural Networks: A Review of Methods and Applications

Surveys graph neural networks-models that capture graph dependencies via message passing-reviewing methods, applications, and open problems.

This survey reviews graph neural networks (GNNs), models capturing dependencies in graph data via message passing between nodes. Many tasks-modeling physical systems, learning molecular fingerprints, or reasoning over dependency trees and scene graphs-need graph inputs; unlike standard networks, GNNs retain neighborhood information at arbitrary depth. Early GNNs were hard to train, but advances in architecture and optimization enabled learning, with variants like GCN, GAT, and gated GNNs excelling. It reviews models, categorizes applications, and poses four open problems.

Based on: Graph Neural Networks: A Review of Methods and Applications · AI Open

Curated by Aramai Editorial

Read summary →
Highlight

CARLA: An Open Urban Driving Simulator

Introduces CARLA, an open-source simulator for autonomous urban driving research with open digital assets and flexible sensor and environment setups.

The authors introduce CARLA, an open-source simulator built to support development, training, and validation of autonomous urban driving systems. Beyond open code and protocols, CARLA provides freely usable digital assets such as urban layouts, buildings, and vehicles, and supports flexible specification of sensor suites and environmental conditions. Using CARLA, they study three approaches to driving: a classic modular pipeline, an end-to-end model trained by imitation learning, and one trained by reinforcement learning, evaluated in scenarios of increasing difficulty.

Based on: CARLA: An Open Urban Driving Simulator · Conference on Robot Learning

Curated by Aramai Editorial

Read summary →
Highlight

Neural Collaborative Filtering

Proposes NCF, a neural framework modeling user-item interactions from implicit feedback by replacing matrix factorization's inner product with a network.

Neural Collaborative Filtering (NCF) applies deep neural networks to collaborative filtering on implicit feedback, the core recommendation problem prior deep-learning work mostly left to matrix factorization. Earlier methods modeled user-item interaction with an inner product over latent features; NCF instead learns an arbitrary interaction function from data, using a multi-layer perceptron for non-linearities, and can generalize matrix factorization. On two real-world datasets it significantly outperforms state-of-the-art methods, and deeper networks improve performance.

Based on: Neural Collaborative Filtering · The Web Conference

Curated by Aramai Editorial

Read summary →
Highlight

Efficient Memory Management for Large Language Model Serving with PagedAttention

Introduces PagedAttention, an OS-paging-inspired attention algorithm, and vLLM, an LLM serving system that cuts KV cache waste and boosts throughput.

High-throughput LLM serving needs batching many requests, but each request's key-value (KV) cache is large and grows and shrinks dynamically, wasting memory via fragmentation and duplication. The authors propose PagedAttention, an attention algorithm inspired by virtual memory and paging in operating systems, and build vLLM on it, achieving near-zero KV cache waste and flexible cache sharing across requests. vLLM improves throughput of popular LLMs by 2-4x at similar latency versus systems like FasterTransformer and Orca, more so for longer sequences and bigger models.

Based on: Efficient Memory Management for Large Language Model Serving with PagedAttention · Symposium on Operating Systems Principles

Curated by Aramai Editorial

Read summary →
Highlight

Character-level Convolutional Networks for Text Classification

Empirically studies character-level convolutional networks for text classification, benchmarked against bag-of-words, n-grams, and word-based models.

This paper empirically explores using character-level convolutional networks (ConvNets) for text classification. The authors construct several large-scale datasets to demonstrate that character-level ConvNets can achieve state-of-the-art or competitive results. They compare against traditional models such as bag-of-words, n-grams, and their TF-IDF variants, as well as deep learning models including word-based ConvNets and recurrent neural networks.

Based on: Character-level Convolutional Networks for Text Classification · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

An overview of gradient descent optimization algorithms

Proposes a statistical test to compare ML models' metric values by splitting the test set into N parts and applying a modified Student's t-test.

The paper proposes a statistical test to compare the metric values of ML models on a test set. Because a metric depends on model and data, different models can win on different test sets, and cross-validation cannot guarantee independent measurements, precluding a t-test. For additive metrics, the author splits the test set into N parts; by the central limit theorem each part's metric is ~normal, so a modified t-test compares means, with normality tests and Q-Q plots sizing samples. A simpler confidence-interval variant proved rougher for CTR cross-entropy.

Based on: An overview of gradient descent optimization algorithms · Vestnik komp iuternykh i informatsionnykh tekhnologii

Curated by Aramai Editorial

Read summary →
Highlight

Adding Conditional Control to Text-to-Image Diffusion Models

Introduces ControlNet, an architecture that adds spatial conditioning (edges, depth, pose, etc.) to pretrained text-to-image diffusion models.

ControlNet is a neural architecture that adds spatial conditioning to large pretrained text-to-image diffusion models. It locks the pretrained model and reuses its encoding layers, learned from billions of images, linking trainable copies via 'zero convolutions'-zero-initialized layers that protect finetuning from noise. Tested controls include edges, depth, segmentation, and human pose on Stable Diffusion, with single or multiple conditions, with or without prompts. Training stays robust on small (<50k) and large (>1m) datasets, broadening control over image diffusion.

Based on: Adding Conditional Control to Text-to-Image Diffusion Models · IEEE International Conference on Computer Vision

Curated by Aramai Editorial

Read summary →
Highlight

Unsupervised Domain Adaptation by Backpropagation

A domain adaptation method learning features discriminative for the source task yet invariant to domain shift, using a simple gradient reversal layer.

Top deep models need massive labeled data, often unavailable for a target task; domain adaptation helps when labeled data from a similar but different domain exists. The authors propose a deep domain adaptation approach trained on labeled source data and unlabeled target data, needing none for the target. It yields features both discriminative for the source task and domain-invariant, via a few standard layers plus a simple gradient reversal layer trained by standard backpropagation. It outperforms prior state of the art on Office datasets under big domain shifts.

Based on: Unsupervised Domain Adaptation by Backpropagation · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →
Highlight

Gaussian Error Linear Units (GELUs)

Proposes the Gaussian Error Linear Unit (GELU), an activation xΦ(x) that weights inputs by their value rather than gating by sign as in ReLUs.

The authors propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function defined as xΦ(x), where Φ(x) is the standard Gaussian cumulative distribution function. Unlike ReLUs, which gate inputs by their sign, GELU weights inputs by their value. An empirical evaluation against the ReLU and ELU activations finds performance improvements across all considered computer vision, natural language processing, and speech tasks.

Based on: Gaussian Error Linear Units (GELUs) · arXiv

Curated by Aramai Editorial

Read summary →
Highlight

Neural Ordinary Differential Equations

Introduces neural ODEs, continuous-depth models that parameterize a hidden state's derivative with a network solved by a black-box ODE solver.

The paper introduces continuous-depth models that parameterize the hidden state's derivative with a neural network and compute outputs via a black-box ODE solver. These models have constant memory cost, adapt computation per input, and trade precision for speed. They demonstrate continuous-depth residual networks and continuous-time latent-variable models, and continuous normalizing flows for maximum-likelihood generative modeling without ordering data dimensions. Gradients backpropagate through any ODE solver without accessing its internals for end-to-end training.

Based on: Neural Ordinary Differential Equations · Neural Information Processing Systems

Curated by Aramai Editorial

Read summary →
Highlight

The STRING database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest

Reports STRING v12.0, a database integrating physical and functional protein-protein associations with new tools for any sequenced genome.

The STRING database collects and integrates protein-protein interactions, spanning physical and functional associations from text mining, co-expression, genomic context, and curated pathways. Scored interactions are transferred to less-studied organisms via orthology and accessed by website, API, or bulk download. Version 12.0 adds interaction-network construction for any novel genome from user-submitted proteins, variational autoencoders with single-cell RNA-seq and proteomics in the co-expression channel, and per-interaction confidence based on detection method.

Based on: The STRING database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest · Nucleic Acids Res.

Curated by Aramai Editorial

Read summary →
Highlight

Addressing Function Approximation Error in Actor-Critic Methods

Proposes methods to curb overestimation bias in actor-critic RL via clipped double Q-learning, delayed policy updates, and target networks.

In value-based RL like deep Q-learning, function approximation errors cause overestimated values and suboptimal policies. The authors show this problem also persists in actor-critic settings and propose mechanisms to reduce its effect on both actor and critic. Building on Double Q-learning, they take the minimum of a pair of critics to limit overestimation, connect target networks to overestimation bias, and delay policy updates to cut per-update error. Evaluated on the OpenAI Gym suite, the method outperforms the state of the art in every environment tested.

Based on: Addressing Function Approximation Error in Actor-Critic Methods · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →