Highlight

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Introduces the Switch Transformer, a simplified sparse mixture-of-experts model that scales to trillion parameters at constant compute cost.

Unlike standard models that reuse the same parameters for all inputs, Mixture of Experts (MoE) selects different parameters per example, giving huge, sparsely activated models at constant compute. Adoption has been limited by complexity, communication cost, and training instability, which the Switch Transformer addresses by simplifying routing and reducing overheads. New techniques tame instabilities and enable bfloat16 training. Based on T5, it delivers up to 7x faster pre-training, gains across 101 languages, and trillion-parameter models with a 4x speedup over T5-XXL.

Based on: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Journal of machine learning research

Curated by Aramai Editorial

Read summary →
Highlight

Random Erasing Data Augmentation

Introduces Random Erasing, a simple CNN data augmentation that randomly masks a rectangular image region to improve robustness to occlusion.

The paper introduces Random Erasing, a data augmentation method for training CNNs. During training it randomly selects a rectangular region in an image and replaces its pixels with random values, generating images with varying occlusion levels that reduce overfitting and make models robust to occlusion. The method needs no learned parameters, is easy to implement, and works with most CNN recognition models. Complementary to random cropping and flipping, it yields consistent gains over strong baselines in image classification, object detection, and person re-identification.

Based on: Random Erasing Data Augmentation · AAAI Conference on Artificial Intelligence

Curated by Aramai Editorial

Read summary →
Highlight

Hyperledger fabric: a distributed operating system for permissioned blockchains

Presents Fabric, a modular open-source platform for permissioned blockchains that runs distributed apps written in general-purpose languages.

Fabric is a modular, extensible open-source system for deploying and operating permissioned blockchains, one of the Linux Foundation's Hyperledger projects. It supports pluggable consensus tailored to specific use cases and trust models, and is the first blockchain to run distributed applications in standard, general-purpose languages with no native cryptocurrency. It handles membership through a portable notion tied to industry identity management. Benchmarked on a Bitcoin-inspired currency, it exceeds 3500 tx/s at sub-second latency, scaling past 100 peers.

Based on: Hyperledger fabric: a distributed operating system for permissioned blockchains · European Conference on Computer Systems

Curated by Aramai Editorial

Read summary →
Highlight

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Improves rectified flow training with perceptually biased noise sampling and a new dual-modality transformer for high-resolution text-to-image synthesis.

Diffusion models generate data by inverting a path from data to noise; rectified flow instead links data and noise on a straight line, with appealing theory but limited adoption. The authors improve rectified flow training by biasing noise sampling toward perceptually relevant scales, and a large-scale study shows it beats established diffusion methods for high-resolution text-to-image synthesis. Their transformer uses separate weights per modality with bidirectional image-text flow, improving text comprehension and typography, and surpasses state-of-the-art models.

Based on: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis · International Conference on Machine Learning

Curated by Aramai Editorial

Read summary →
Highlight

Deep learning in agriculture: A survey

Surveys 40 research efforts applying deep learning to agricultural and food production problems, comparing models, data, and performance.

Deep learning is a modern technique for image processing and data analysis that has recently entered agriculture. This survey reviews 40 research efforts applying deep learning to various agricultural and food production challenges. It examines the problems studied, the models and frameworks used, data sources, nature and preprocessing, and the performance achieved under each work's metrics, and compares deep learning with other popular techniques. The findings indicate deep learning delivers high accuracy, outperforming commonly used image processing methods.

Based on: Deep learning in agriculture: A survey · Computers and Electronics in Agriculture

Curated by Aramai Editorial

Read summary →
Highlight

Scaling Instruction-Finetuned Language Models

Studies instruction finetuning at scale across more tasks, larger models, and chain-of-thought data, yielding Flan-PaLM and released Flan-T5 checkpoints.

Finetuning language models on datasets phrased as instructions improves performance and generalization to unseen tasks. This work scales instruction finetuning along three axes: number of tasks, model size, and chain-of-thought data. Across model classes (PaLM, T5, U-PaLM), prompting setups, and benchmarks, it sharply improves results: Flan-PaLM 540B tuned on 1.8K tasks beats PaLM 540B by +9.4% on average and hits 75.2% on five-shot MMLU. The released Flan-T5 checkpoints show strong few-shot performance even against much larger models.

Based on: Scaling Instruction-Finetuned Language Models · Journal of machine learning research

Curated by Aramai Editorial

Read summary →
Highlight

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Investigates an early version of GPT-4, arguing it shows more general intelligence than prior models across many domains and tasks.

The paper reports an investigation of an early, still-in-development version of OpenAI's GPT-4, trained at unprecedented scale. The authors argue it belongs to a new cohort of LLMs with more general intelligence than earlier AI, solving novel, hard tasks across mathematics, coding, vision, medicine, law, and psychology without special prompting, often near or beyond human level. They suggest it is an early, incomplete form of AGI, stress its limitations, and discuss challenges ahead, including moving beyond next-word prediction.

Based on: Sparks of Artificial General Intelligence: Early experiments with GPT-4 · arXiv.org

Curated by Aramai Editorial

Read summary →
Highlight

Pointer Sentinel Mixture Models

Introduces the pointer sentinel mixture architecture that lets neural sequence models copy words from recent context or use a softmax classifier.

Softmax neural sequence models reach top language modeling only with large hidden states and vocabularies, yet still fail on rare or unseen words even when context is unambiguous. The authors propose a pointer sentinel mixture that either reproduces a word from recent context or generates one via a standard softmax. Their pointer sentinel-LSTM reaches 70.9 perplexity on Penn Treebank using far fewer parameters, and they release the WikiText corpus for evaluating longer contexts and larger vocabularies.

Based on: Pointer Sentinel Mixture Models · International Conference on Learning Representations

Curated by Aramai Editorial

Read summary →
Highlight

Guidelines for conducting and reporting case study research in software engineering

Provides an introduction and practical guidelines, including checklists, for conducting and reporting case study research in software engineering.

Case study research suits software engineering because it studies contemporary phenomena in their natural context, yet what counts as a case study varies, and so does study quality. This paper introduces case study methodology and offers guidelines for researchers who conduct them and readers who assess such reports. Drawing on the authors' experience, it adapts terminology from handbooks in social science and information systems to software engineering, and presents recommended practices plus empirically derived, evaluated checklists for researchers and readers.

Based on: Guidelines for conducting and reporting case study research in software engineering · Empirical Software Engineering

Curated by Aramai Editorial

Read summary →
Highlight

Randaugment: Practical automated data augmentation with a reduced search space

Introduces RandAugment, a data augmentation method with a greatly reduced search space that removes the need for a separate proxy search phase.

Automated data augmentation yields state-of-the-art results but usually needs a separate, expensive search phase, often run on a proxy task whose hyperparameters may not transfer. The authors find that instead of independently searching each operation's magnitude and probability, a single distortion magnitude jointly controlling all operations suffices. This simplified space cuts cost and removes the proxy task, yet matches or beats prior methods on CIFAR-10/100, SVHN, ImageNet, and COCO, reaching 85.0% ImageNet accuracy with EfficientNet-B7 and 85.4% with B8.

Based on: Randaugment: Practical automated data augmentation with a reduced search space · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Curated by Aramai Editorial

Read summary →
Highlight

A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects

Surveys convolutional neural networks, covering their history, 1-D/2-D/multidimensional convolutions, key models, practical tips, and open issues.

Convolutional neural networks (CNNs) are among the most important deep-learning models, impacting computer vision, NLP, and more. Noting that prior reviews focus on applications rather than a general perspective and omit recent ideas, this survey offers a broader view spanning 1-D, 2-D, and multidimensional convolutions. It traces CNN history, overviews convolution types, introduces classic and advanced models and their key ideas, and derives rules of thumb for functions and hyperparameters via experiments, before reviewing applications and open future directions.

Based on: A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects · IEEE Transactions on Neural Networks and Learning Systems

Curated by Aramai Editorial

Read summary →
Highlight

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Proposes temporal segment networks (TSN) with sparse sampling and video-level supervision for effective deep action recognition in video.

Deep convolutional networks dominate still-image recognition, but their advantage for video action recognition has been less clear. This paper seeks principles for designing effective ConvNet architectures that learn from limited data. Its main contribution is the temporal segment network (TSN), which models long-range temporal structure by combining sparse temporal sampling with video-level supervision to learn from whole videos. The authors also study good practices for training ConvNets on video, achieving state-of-the-art results on HMDB51 (69.4%) and UCF101 (94.2%).

Based on: Temporal Segment Networks: Towards Good Practices for Deep Action Recognition · European Conference on Computer Vision

Curated by Aramai Editorial

Read summary →