Enhancing Deep Learning Performance Using Displaced Rectifier Linear Unit
By David Macêdo
Length1h 49m
About this audiobook
Recently, deep learning has caused a significant impact on computer vision, speech recognition, and natural language understanding. In spite of the remarkable advances, deep learning recent performance gains have been modest and usually rely on increasing the depth of the models, which often requires more computational resources such as processing time and memory usage. To tackle this problem, we turned our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory currently. In this work, we propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-10 and CIFAR-100, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy obtained by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception.
Audiobook details
GenreTechnology
Length1 hr 49 mins
Narrated byListen with 1,000+ voices
FormateBook with Audio
Publish dateMar 25, 2022
LanguageEnglish
Table of contents
1Acknowledgements
224. Experiments
21. Introduction
234.1 DATASETS, PREPROCESSING AND DATA AUGMENTATION
31.1 CONTEXT
244.2 ACTIVATION FUNCTIONS PARAMETRIZATION
41.2 PROBLEM
254.3 MODELS AND INITIALIZATION
51.3 GOAL
264.4 TRAINING AND REGULARIZATION
Show all chaptersShow less
61.4 OUTLINE
274.5 PERFORMANCE ASSESSMENT
72. Background
285. Results
82.1 DEEP LEARNING
295.1 BIAS SHIFT EFFECT
92.2 ACTIVATION FUNCTIONS
305.2 CIFAR-10 DATASET
102.2.1 Rectifier Linear Unit
315.2.1 VGG-19 Model
112.2.2 Leaky Rectifier Linear Unit
325.2.2 ResNet-56 Model
122.2.3 Parametric Rectifier Linear Unit
335.2.3 ResNet-110 Model
132.2.4 Exponential Linear Unit
345.3 CIFAR-100 DATASET
142.3 CONVOLUTIONAL NETWORKS
355.3.1 VGG-19 Model
152.4 ARCHITECTURES
365.3.2 ResNet-56 Model
162.4.1 Visual Geometry Group
375.3.3 ResNet-110 Model
172.4.2 Residual Networks
385.4 DISCUSSION
182.5 REGULARIZATION
396. Conclusion
192.5.1 Dropout
406.1 CONTRIBUTIONS
202.5.2 Batch Normalization
416.2 FUTURE WORK
213. Displaced Rectifier Linear Unit
Never Lost AgainBill Kilday10h 1m4.4 (805)$29 · $0.00
How the Internet HappenedBrian McCullough13h 28m4.3 (2.7K)$23 · $0.00
Collected Writings of Nikola TeslaNikola Tesla, Thomas Commerford Martin21h 50m4.3 (1.7K)$1 · $0.00
Just AspireAjai Chowdhry9h 13m4.4 (260)$29 · $0.00
The Little Book of AliensAdam Frank8h 20m4.3 (1.3K)$26 · $0.00
The inventions, researches and writings of Nikola Tesla (Annotated)Thomas Commerford Martin16h 59m4.3 (1.3K)$2 · $0.00
KargilV.P. Malik15h 18m4.3 (2K)$24 · $0.00
Experiments with Alternating CurrentsNikola Tesla9h 16m4.3 (874)$0.99 · $0.00
Fire on the HorizonTom Shroder, John Konrad8h 23m4.3 (858)$26 · $0.00
The Smell of Kerosene (Annotated)National Aeronautics and Space Administration, Donald L. Mallick, Peter W. Merlin11h 32m4.3 (523)$2 · $0.00
The Boy Who Harnessed the WindWilliam Kamkwamba, Bryan Mealer10h 5m4.2 (78.9K)$29 · $0.00
Across the Airless WildsEarl Swift10h 8m4.3 (1.3K)$29 · $0.00
Console WarsBlake J. Harris20h 41m4.2 (13.7K)$46 · $0.00
Improvised Munitions HandbookU.S. Department of Defense4h 49m4.3 (1.6K)$1.99 · $0.00
Escaping GravityLori Garver10h 54m4.2 (2.2K)$20 · $0.00
Wireless WarsJonathan Pelson8h 2m4.3 (419)$20 · $0.00
Let There Be WaterSeth M. Siegel8h 36m4.2 (2.3K)$20 · $0.00
The PerfectionistsSimon Winchester11h 46m4.2 (9.9K)$29 · $0.00
Rethinking NarcissismDr. Craig Malkin6h 7m4.2 (3.4K)$24 · $0.00
Project Animal FarmSonia Faruqi11h 23m4.3 (563)$20 · $0.00
You may also like
ITIL® 4 Create, Deliver and Support (CDS)Claire Agutter3h 36m4.6 (23)$40 · $0.00
ITIL® 4 Essentials: Your essential guide for the ITIL 4 Foundation exam and beyond, second editionClaire Agutter4h4.2 (50)$40 · $0.00
Algorithms in Java, Coded and TestedBianca Zielinski7h 25m$5 · $0.00
Machine Learning in the CloudLionel Jansen8h 21m$5 · $0.00
The AI Advantage for Restaurant OwnersMiles Draper4h 21m$5
Build a 2.5D RPG with Godot 4 and C#Gregory Whitlock7h 8m$5 · $0.00
Bio-Inspired AI in C++Zoe Becker7h 10m$5 · $0.00
Python Programming Made PracticalAustin Gray7h 21m5.0 (1)$6.50 · $0.00
Ethical Hacking Practice LabAaron Blake7h 51m$6.50 · $0.00
The AI Advantage for LandlordsMiles Draper3h 59m$9
Smart TV Guide for BeginnersWalter Hemsley4h 12m$7
Claude AI Prompts for Finance ProfessionalsMiles Draper4h 20m$10
ElevenLabs Made SimpleMiles Draper4h 27m$6
A High-Performance Adaptive Filter Instantiated from the GAADT Algorithm for Electrocardiogram Signal ProcessingAndrilene Ferreira Maciel2h 55m$13 · $0.00
Monetize your content with NFTsUnknown Author1h 5m$17 · $0.00
Statistical Methods for Engineers and ScientistsPriya Ramanathan15h 5m$95
THE CLAUDE AI BIBLE FOR BEGINNERSSheldon Caldwell4h 42m$17.50 · $0.00
How to Profit from Google AdSenseAnthony Ekanem37m$4 · $0.00
From Zero To AutomaticSara Melo1h 25m$1.80 · $0.00
n8n Automation Made SimpleMiles Draper3h 54m$7