Enhancing Deep Learning Performance Using Displaced Rectifier Linear Unit
By David Macêdo
Length1h 49m
About this audiobook
Recently, deep learning has caused a significant impact on computer vision, speech recognition, and natural language understanding. In spite of the remarkable advances, deep learning recent performance gains have been modest and usually rely on increasing the depth of the models, which often requires more computational resources such as processing time and memory usage. To tackle this problem, we turned our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory currently. In this work, we propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-10 and CIFAR-100, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy obtained by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception.
Audiobook details
GenreTechnology
Length1 hr 49 mins
Narrated byListen with 1,000+ voices
FormateBook with Audio
Publish dateMar 25, 2022
LanguageEnglish
Table of contents
1Acknowledgements
224. Experiments
21. Introduction
234.1 DATASETS, PREPROCESSING AND DATA AUGMENTATION
31.1 CONTEXT
244.2 ACTIVATION FUNCTIONS PARAMETRIZATION
41.2 PROBLEM
254.3 MODELS AND INITIALIZATION
51.3 GOAL
264.4 TRAINING AND REGULARIZATION
Mostrar todos os capítulosVer menos
61.4 OUTLINE
274.5 PERFORMANCE ASSESSMENT
72. Background
285. Results
82.1 DEEP LEARNING
295.1 BIAS SHIFT EFFECT
92.2 ACTIVATION FUNCTIONS
305.2 CIFAR-10 DATASET
102.2.1 Rectifier Linear Unit
315.2.1 VGG-19 Model
112.2.2 Leaky Rectifier Linear Unit
325.2.2 ResNet-56 Model
122.2.3 Parametric Rectifier Linear Unit
335.2.3 ResNet-110 Model
132.2.4 Exponential Linear Unit
345.3 CIFAR-100 DATASET
142.3 CONVOLUTIONAL NETWORKS
355.3.1 VGG-19 Model
152.4 ARCHITECTURES
365.3.2 ResNet-56 Model
162.4.1 Visual Geometry Group
375.3.3 ResNet-110 Model
172.4.2 Residual Networks
385.4 DISCUSSION
182.5 REGULARIZATION
396. Conclusion
192.5.1 Dropout
406.1 CONTRIBUTIONS
202.5.2 Batch Normalization
416.2 FUTURE WORK
213. Displaced Rectifier Linear Unit
When the Heavens Went on SaleAshlee Vance18h 20m4.4 (3K)$40 · $0.00
Everybody Has a Podcast (Except You)Justin McElroy, Travis McElroy, Griffin McElroy5h 9m4.3 (2.3K)$24 · $0.00
Hands of TimeRebecca Struthers8h 8m4.4 (1.3K)$26 · $0.00
Never Lost AgainBill Kilday10h 1m4.4 (805)$29 · $0.00
How the Internet HappenedBrian McCullough13h 28m4.3 (2.7K)$23 · $0.00
Collected Writings of Nikola TeslaNikola Tesla, Thomas Commerford Martin21h 50m4.3 (1.7K)$1 · $0.00
Just AspireAjai Chowdhry9h 13m4.4 (260)$29 · $0.00
The Little Book of AliensAdam Frank8h 20m4.3 (1.3K)$26 · $0.00
The inventions, researches and writings of Nikola Tesla (Annotated)Thomas Commerford Martin16h 59m4.3 (1.3K)$2 · $0.00
KargilV.P. Malik15h 18m4.3 (2K)$24 · $0.00
Experiments with Alternating CurrentsNikola Tesla9h 16m4.3 (874)$0.99 · $0.00
Fire on the HorizonTom Shroder, John Konrad8h 23m4.3 (858)$26 · $0.00
The Smell of Kerosene (Annotated)National Aeronautics and Space Administration, Donald L. Mallick, Peter W. Merlin11h 32m4.3 (523)$2 · $0.00
The Boy Who Harnessed the WindWilliam Kamkwamba, Bryan Mealer10h 5m4.2 (78.9K)$29 · $0.00
Across the Airless WildsEarl Swift10h 8m4.3 (1.3K)$29 · $0.00
Console WarsBlake J. Harris20h 41m4.2 (13.7K)$46 · $0.00
Improvised Munitions HandbookU.S. Department of Defense4h 49m4.3 (1.6K)$1.99 · $0.00
Escaping GravityLori Garver10h 54m4.2 (2.2K)$20 · $0.00
Wireless WarsJonathan Pelson8h 2m4.3 (419)$20 · $0.00
Let There Be WaterSeth M. Siegel8h 36m4.2 (2.3K)$20 · $0.00
You may also like
Collected Writings of Nikola TeslaThomas Commerford Martin, Nikola Tesla19h 42m4.4 (2.8K)$0.99 · $0.00
Making SenseSam Harris22h 27m4.1 (3K)$46 · $0.00
WTF?Tim O'Reilly16h 15m4.2 (2.6K)$33 · $0.00
AI ValleyGary Rivlin11h 18m4.1 (452)$29 · $0.00
The NumeratiStephen Baker7h 55m3.6 (2K)$17 · $0.00
The AI ConEmily M. Bender, Alex Hanna7h 51m3.7 (1.7K)$26 · $0.00
The Highway Code UKDepartment for Transport3h 33m4.2 (36.6K)$11 · $0.00
Machines of Loving GraceJohn Markoff11h 53m3.8 (1.4K)$29 · $0.00
How To Code For Quantum ComputersNivio Dos Santos2h 19m5.0 (1)$46 · $0.00
ScaleBradley J Sugars3h 27m4.6 (52)Free
Smarter Games with AICurtis Whitlock7h 25m$5 · $0.00
Social Media And Online BusinessEdivaldo Ascaneo1h 25m$6 · $0.00
Brazilian Portuguese for Beginners: The Complete A1 Audio CourseMichele Cavalcante12h 5m$32 · $0.00
Quick Brazilian PortugueseJack Scholes22m$1 · $0.00
Englisch lernen Audio - Die ZeitformenSpotlight Verlag1h 2m4.4 (14)$17
Business-Englisch lernen Audio - Strategisch handelnSpotlight Verlag1h 12m$19
The Reverse Centaur's Guide to ThinkingKyle Patterson2h 37m$14 · $0.00
Leads MachineFrancis Rodino6h 30m4.9 (39)$27 · $0.00
Passages from the Life of a Philosopher (Annotated)Charles Babbage16h 33m4.4 (110)$1.99 · $0.00
The Nightlight with Melvin VallomparambilChristopher Glyn36m$4 · $0.00