arXiv:1707.07012v3 [cs.CV] 1 Dec 2017
Learning Transferable Architectures for Scalable Image Recognition Barret Zoph Google Brain
Vijay Vasudevan Google Brain
Jonathon Shlens Google Brain
Quoc V. Le Google Brain
[email protected]
[email protected]
[email protected]
[email protected]e.com
Abstract
cation represents one of the most important breakthroughs in deep learning. Successive advancements on this benchmark based on convolutional neural networks (CNNs) have achieved impressive results through significant architecture engineering [52, 58, 20, 59, 57, 67].
Developing neural network image classification models often requires significant architecture engineering. In this paper, we attempt to automate this engineering process by learning the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. Our key contribution is the design of a new search space which enables transferability. In our experiments, we search for the best convolutional layer (or “cell”) on the CIFAR10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters. Although the cell is not searched for directly on ImageNet, an architecture constructed from the best cell achieves, among the published works, stateoftheart accuracy of 82.7% top1 and 96.2% top5 on ImageNet. Our model is 1.2% better in top1 accuracy than the best humaninvented architectures while having 9 billion fewer FLOPS – a reduction of 28% in computational demand from the previous stateoftheart model. When evaluated at different levels of computational cost, accuracies of our models exceed those of the stateoftheart humandesigned models. For instance, a smaller network constructed from the best cell also achieves 74% top1 accuracy, which is 3.1% better than equivalentlysized, stateoftheart models for mobile platforms. On CIFAR10, an architecture constructed from the best cell achieves 2.4% error rate, which is also stateoftheart. Finally, the image features learned from image classification can also be transferred to other computer vision problems. On the task of object detection, the learned features used with the FasterRCNN framework surpass stateoftheart by 4.0% achieving 43.1% mAP on the COCO dataset.
In this paper, we consider learning the convolutional architectures directly from data with application to ImageNet classification. In addition to being an difficult and important benchmark in computer vision, features derived from ImageNet classifiers are of great importance to many other computer vision tasks. For example, features from networks that perform well on ImageNet classification provide stateoftheart performance when transferred to other computer vision tasks where labeled data is limited [13].
1. Introduction
Our approach is inspired by the recently proposed Neural Architecture Search (NAS) framework [70], which uses a policy gradient algorithm to optimize architecture configurations. Even though NAS is attractive as a method to search for good convolutional network architectures, applying it directly to the ImageNet dataset is computationally expensive given the size of the dataset. We therefore propose to search for a good architecture on the far smaller CIFAR10 dataset, and automatically transfer the learned architecture to ImageNet. We achieve this transferrability by designing a search space so that the complexity of the architecture is independent of the depth of the network and the size of input images. More concretely, all convolutional networks in our search space are composed of convolutional layers (or “cells”) with identical structure but different weights. Searching for the best convolutional architectures is therefore reduced to searching for the best cell structure. Searching for the best cell structure has two main benefits: it is much faster than searching for an entire network architecture and the cell itself is more likely to generalize to other problems. In our experiments, this approach significantly accelerates the search for the best architectures using CIFAR10 by a factor of 7× and learns architectures that successfully transfer to ImageNet.
ImageNet classification [11] is an important benchmark in computer vision. The seminal work of [32] on using convolutional architectures [17, 33] for ImageNet classifi
Our main result is that the best architecture found on CIFAR10 achieves stateoftheart accuracy when transferred to ImageNet classification without much modifica1
tion. On ImageNet, an architecture constructed from the best cell achieves, among the published works, stateoftheart accuracy of 82.7% top1 and 96.2% top5. This result amounts to a 1.2% improvement in top1 accuracy than the best humaninvented architectures while having 9 billion fewer FLOPS. On CIFAR10 itself, the architecture achieves 2.4% error rate, which is also stateoftheart. Additionally, by simply varying the number of the convolutional cells and number of filters in the convolutional cells, we can create convolutional architectures with different computational demands. Thanks to this property of the cells, we can generate a family of models that achieve accuracies superior to all humaninvented models at equivalent or smaller computational budgets [59, 29]. Notably, the smallest version of the learned model achieves 74.0% top1 accuracy on ImageNet, which is 3.1% better than previously engineered architectures targeted towards mobile and embedded vision tasks [24, 69]. Finally, we show that the image features learned from image classification are generically useful and transfer to other computer vision problems. In our experiments, the features learned from ImageNet classification can be combined with the FasterRCNN framework [46] to achieve stateoftheart on COCO object detection task for both the largest as well as mobileoptimized models. Our largest model achieves 43.1% mAP, which is 4% better than previous stateoftheart.
2. Method Our work makes use of search methods to find good convolutional architectures on a dataset of interest. The main search method we use in this work is the Neural Architecture Search (NAS) framework proposed by [70]. In NAS, a controller recurrent neural network (RNN) samples child networks with different architectures. The child networks are trained to convergence to obtain some accuracy on a heldout validation set. The resulting accuracies are used to update the controller so that the controller will generate better architectures over time. The controller weights are updated with policy gradient (see Figure 1). The main contribution of this work is the design of a novel search space, such that the best architecture found on the CIFAR10 dataset would scale to larger, higherresolution image datasets across a range of computational settings. One inspiration for this search space is the recognition that architecture engineering with CNNs often identifies repeated motifs consisting of combinations of convolutional filter banks, nonlinearities and a prudent selection of connections to achieve stateoftheart results (such as the repeated modules present in the Inception and ResNet models [58, 20, 59, 57]). These observations suggest that it may be possible for the controller RNN to predict a generic convolutional cell expressed in terms of these motifs. This cell
Sample architecture A! with probability p
Train a child network! with architecture A to ! convergence to get ! validation accuracy R
The controller (RNN)
Scale gradient of p by R! to update the controller
Figure 1. Overview of Neural Architecture Search [70]. A controller RNN predicts architecture A from a search space with probability p. A child network with architecture A is trained to convergence achieving accuracy R. Scale the gradients of p by R to update the RNN controller.
can then be stacked in series to handle inputs of arbitrary spatial dimensions and filter depth. In our approach, the overall architectures of the convolutional nets are manually predetermined. They are composed of convolutional cells repeated many times where each convolutional cell has the same architecture, but different weights. To easily build scalable architectures for images of any size, we need two types of convolutional cells to serve two main functions when taking in a feature map as input: (1) convolutional cells that return a feature map of the same dimension, and (2) convolutional cells that return a feature map where the feature map height and width is reduced by a factor of two. We name the first type and second type of convolutional cells Normal Cell and Reduction Cell respectively. For the Reduction Cell, we make the initial operation applied to the cell’s inputs have a stride of two to reduce the height and width. All of our operations that we consider for building our convolutional cells have an option of striding. Figure 2 shows our placement of Normal and Reduction Cells for CIFAR10 and ImageNet. Note on ImageNet we have more Reduction Cells, since the incoming image size is 299x299 compared to 32x32 for CIFAR. The Reduction and Normal Cell could have the same architecture, but we empirically found it beneficial to learn two separate architectures. We use a common heuristic to double the number of filters in the output whenever the spatial activation size is reduced in order to maintain roughly constant hidden state dimension [32, 52]. Importantly, much like Inception and ResNet models [58, 20, 59, 57], we consider the number of motif repetitions N and the number of initial convolutional filters as free parameters that we tailor to the scale of an image classification problem. What varies in the convolutional nets is the structures of the Normal and Reduction Cells, which are searched by the controller RNN. The structures of the cells can be searched
provides good results, although we have not exhaustively searched this space due to computational limitations. In steps 3 and 4, the controller RNN selects an operation to apply to the hidden states. We collected the following set of operations based on their prevalence in the CNN literature: • • • • • • •
Figure 2. Scalable architectures for image classification consist of two repeated motifs termed Normal Cell and Reduction Cell. This diagram highlights the model architecture for CIFAR10 and ImageNet. The choice for the number of times the Normal Cells that gets stacked between reduction cells, N , can vary in our experiments.
within a search space defined as follows. In our search space, each cell receives as input two initial hidden states hi and hi−1 which are the outputs of two cells in previous two lower layers or the input image. The controller RNN recursively predicts the rest of the structure of the convolutional cell, given these two initial hidden states (Figure 3). The predictions of the controller for each cell are grouped into B blocks, where each block has 5 prediction steps made by 5 distinct softmax classifiers corresponding to discrete choices of the elements of a block: Step 1. Select a hidden state from hi , hi−1 or from the set of hidden states created in previous blocks. Step 2. Select a second hidden state from the same options as in Step 1. Step 3. Select an operation to apply to the hidden state selected in Step 1. Step 4. Select an operation to apply to the hidden state selected in Step 2. Step 5. Select a method to combine the outputs of Step 3 and 4 to create a new hidden state.
The algorithm appends the newlycreated hidden state to the set of existing hidden states as a potential input in subsequent blocks. The controller RNN repeats the above 5 prediction steps B times corresponding to the B blocks in a convolutional cell. In our experiments, selecting B = 5
identity 1x7 then 7x1 convolution 3x3 average pooling 5x5 max pooling 1x1 convolution 3x3 depthwiseseparable conv 7x7 depthwiseseparable conv
• • • • • •
1x3 then 3x1 convolution 3x3 dilated convolution 3x3 max pooling 7x7 max pooling 3x3 convolution 5x5 depthwiseseperable conv
In step 5 the controller RNN selects a method to combine the two hidden states, either (1) elementwise addition between two hidden states or (2) concatenation between two hidden states along the filter dimension. Finally, all of the unused hidden states generated in the convolutional cell are concatenated together in depth to provide the final cell output. To allow the controller RNN to predict both Normal Cell and Reduction Cell, we simply make the controller have 2 × 5B predictions in total, where the first 5B predictions are for the Normal Cell and the second 5B predictions are for the Reduction Cell. Finally, our work makes use of the reinforcement learning proposal in NAS [70] intensively; however, it is also possible to use random search to search for models in our search space. In random search, instead of sampling the decisions from the softmax classifiers in the controller RNN, we can sample the decisions from the uniform distribution. In our experiments, we find that random search is worse than reinforcement learning on the CIFAR10 dataset. Although there is value in using reinforcement learning, the gap is smaller than what is found in the original work of [70]. This result suggests that the new search space is well designed such that random search can perform reasonably well. We will compare reinforcement learning against random search in Section 3.4.
3. Experiments and Results In this section, we describe our experiments with our method described above to learn convolutional cells. In summary, all architecture searches are performed using the CIFAR10 classification task [31]. The controller RNN was trained using Proximal Policy Optimization (PPO) [50] by employing a global workqueue system for generating a pool of child networks controlled by the RNN. In our experiments, the pool of workers in the workqueue consisted of 500 GPUs. Please see Appendix A for complete details of the architecture learning algorithm and the controller system.
softmax! layer
Select second! hidden state
Select operation for ! first hidden state
Select operation for! second hidden state
controller! hidden layer
Select one! hidden state
Select method to! combine hidden state
new hidden layer
add
repeat B times
3 x 3 conv
2 x 2 maxpool
hidden layer A
hidden layer B
Figure 3. Controller model architecture for recursively constructing one block of a convolutional cell. Each block requires selecting 5 discrete parameters, each of which corresponds to the output of a softmax layer. Example constructed block shown on right. A convolutional cell contains B blocks, hence the controller contains 5B softmax layers for predicting the architecture of a convolutional cell. In our experiments, the number of blocks B is 5.
The result of this search process over 4 days yields several candidate convolutional cells. We note that this search procedure is almost 7× faster than previous approaches [70] that took 28 days.1 Additionally, we demonstrate below that the resulting architecture is superior in accuracy. Figure 4 shows a diagram of the top performing Normal Cell and Reduction Cell. Note the prevalence of separable convolutions and the number of branches compared with competing architectures [52, 58, 20, 59, 57]. Subsequent experiments focus on this convolutional cell architecture, although we examine the efficacy of other, topranked convolutional cells in ImageNet experiments (described in Appendix B) and report their results as well. We call the three networks constructed from the best three searches NASNetA, NASNetB and NASNetC. We demonstrate the utility of the convolutional cells by employing this learned architecture on CIFAR10 and a family of ImageNet classification tasks. The latter family of tasks is explored across a few orders of magnitude in computational budget. After having learned the convolutional cells, several hyperparameters may be explored to build a final network for a given task: (1) the number of cell repeats N and (2) the number of filters in the initial convolutional cell. After selecting the number of initial filters, we use a common heuristic to double the number of filters whenever the stride is 2. Finally, we define a simple notation, e.g., 4 @ 64, to indicate these two parameters in all networks, where 4 and 64 indicate the number of cell repeats and the number of filters in the penultimate layer of the network, respectively.
1 In particular, we note that previous architecture search [70] used 800 GPUs for 28 days resulting in 22,400 GPUhours. The method in this paper uses 500 GPUs across 4 days resulting in 2,000 GPUhours. The former effort used Nvidia K40 GPUs, whereas the current efforts used faster NVidia P100s. Discounting the fact that the we use faster hardware, we estimate that the current procedure is roughly about 7× more efficient.
3.1. Results on CIFAR10 Image Classification For the task of image classification with CIFAR10, we set N = 4 or 6 (Figure 2). The test accuracies of the best architectures are reported in Table 1 along with other stateoftheart models. As can be seen from the Table, a large NASNetA model with cutout data augmentation [12] achieves a stateoftheart error rate of 2.40% (averaged across 5 runs), which is slightly better than the previous best record of 2.56% by [12]. The best single run from our model achieves 2.19% error rate.
3.2. Results on ImageNet Image Classification We performed several sets of experiments on ImageNet with the best convolutional cells learned from CIFAR10. We emphasize that we merely transfer the architectures from CIFAR10 but train all ImageNet models weights from scratch. Results are summarized in Table 2 and 3 and Figure 5. In the first set of experiments, we train several image classification systems operating on 299x299 or 331x331 resolution images with different experiments scaled in computational demand to create models that are roughly on par in computational cost with Inceptionv2 [29], Inceptionv3 [59] and PolyNet [68]. We show that this family of models achieve stateoftheart performance with fewer floating point operations and parameters than comparable architectures. Second, we demonstrate that by adjusting the scale of the model we can achieve stateoftheart performance at smaller computational budgets, exceeding streamlined CNNs handdesigned for this operating regime [24, 69]. Note we do not have residual connections between convolutional cells as the models learn skip connections on their own. We empirically found manually inserting residual connections between cells to not help performance. Our training setup on ImageNet is similar to [59], but please see Appendix A for details. Table 2 shows that the convolutional cells discovered with CIFAR10 generalize well to ImageNet prob
hi+1
concat
hi+1 add
add
concat max! 3x3 add
sep! 3x3
add
iden! tity
sep! 3x3
add
sep! 5x5
iden! tity
avg! 3x3
add
avg! 3x3
sep! 3x3
add
avg! 3x3
sep! 5x5
avg! 3x3
iden! tity
add
sep! 3x3
sep! 7x7
add
sep! 5x5
max! 3x3
hi
hi
...
...
hi1
hi1
Normal Cell
sep! 7x7
add
avg! 3x3
sep! 5x5
Reduction Cell
Figure 4. Architecture of the best convolutional cells (NASNetA) with B = 5 blocks identified with CIFAR10 . The input (white) is the hidden state from previous activations (or input image). The output (pink) is the result of a concatenation operation across all resulting branches. Each convolutional cell is the result of B blocks. A single block is corresponds to two primitive operations (yellow) and a combination operation (green). Note that colors correspond to operations in Figure 3.
model
depth
# params
error rate (%)
DenseNet (L = 40, k = 12) [26] DenseNet(L = 100, k = 12) [26] DenseNet (L = 100, k = 24) [26] DenseNetBC (L = 100, k = 40) [26]
40 100 100 190
1.0M 7.0M 27.2M 25.6M
5.24 4.10 3.74 3.46
ShakeShake 26 2x32d [18] ShakeShake 26 2x96d [18] ShakeShake 26 2x96d + cutout [12]
26 26 26
2.9M 26.2M 26.2M
3.55 2.86 2.56
NAS v3 [70] NAS v3 [70]
39 39
7.1M 37.4M
4.47 3.65

3.3M 3.3M 27.6M 27.6M 2.6M 3.1M
3.41 2.65 2.97 2.40 3.73 3.59
NASNetA NASNetA NASNetA NASNetA NASNetB NASNetC
(6 @ 768) (6 @ 768) + cutout (7 @ 2304) (7 @ 2304) + cutout (4 @ 1152) (4 @ 640)
Table 1. Performance of Neural Architecture Search and other stateoftheart models on CIFAR10. All results for NASNet are the mean accuracy across 5 runs.
lems. In particular, each model based on the convolutional cells exceeds the predictive performance of the corresponding handdesigned model. Importantly, the largest model achieves a new stateoftheart performance for ImageNet (82.7%) based on single, nonensembled predictions, surpassing previous best published result by ∼1.2% [8]. Among the unpublished works, our model is on par with the best reported result of 82.7% [25], while having significantly fewer floating point operations. Figure 5 shows a complete summary of our results in comparison with other
published results. Note the family of models based on convolutional cells provides an envelope over a broad class of humaninvented architectures. Finally, we test how well the best convolutional cells may perform in a resourceconstrained setting, e.g., mobile devices (Table 3). In these settings, the number of floating point operations is severely constrained and predictive performance must be weighed against latency requirements on a device with limited computational resources. MobileNet [24] and ShuffleNet [69] provide stateoftheart re
85
85
NASNetA (6 @ 4032)
NASNetA (6 @ 4032)
80
PolyNet InceptionResNetv2 NASNetA (5 @ 1538) Inceptionv4 ResNet152 Xception
DPN131 ResNeXt101
NASNetA (7 @ 1920)
SENet
accuracy (precision @1)
accuracy (precision @1)
NASNetA (7 @ 1920)
Inceptionv3
75
Inceptionv2 NASNetA (4 @ 1056) VGG16 ShuffleNet MobileNet Inceptionv1
70
80
DPN131 InceptionResNetv2
Inceptionv3
SENet
PolyNet ResNeXt101
NASNetA (5 @ 1538) Inceptionv4 Xception ResNet152
Inceptionv2
75
NASNetA (4 @ 1056) VGG16 ShuffleNet MobileNet
70
Inceptionv1
65
65 0
10000
20000
30000
0
40000
40
20
# MultAdd operations (millions)
100
80
60
120
140
# parameters (millions)
Figure 5. Accuracy versus computational demand (left) and number of parameters (right) across top performing published CNN architectures on ImageNet 2012 ILSVRC challenge prediction task. Computational demand is measured in the number of floatingpoint multiplyadd operations to process a single image. Black circles indicate previously published results and red squares highlight our proposed models.
Model
image size
# parameters
MultAdds
Top 1 Acc. (%)
Top 5 Acc. (%)
Inception V2 [29] NASNetA (5 @ 1538)
224×224 299×299
11.2 M 10.9 M
1.94 B 2.35 B
74.8 78.6
92.2 94.2
Inception V3 [59] Xception [9] Inception ResNet V2 [57] NASNetA (7 @ 1920)
299×299 299×299 299×299 299×299
23.8 M 22.8 M 55.8 M 22.6 M
5.72 B 8.38 B 13.2 B 4.93 B
78.0 79.0 80.4 80.8
93.9 94.5 95.3 95.3
ResNeXt101 (64 x 4d) [67] PolyNet [68] DPN131 [8] SENet [25] NASNetA (6 @ 4032)
320×320 331×331 320×320 320×320 331×331
83.6 M 92 M 79.5 M 145.8 M 88.9 M
31.5 B 34.7 B 32.0 B 42.3 B 23.8 B
80.9 81.3 81.5 82.7 82.7
95.6 95.8 95.8 96.2 96.2
Table 2. Performance of architecture search and other published stateoftheart models on ImageNet classification. MultAdds indicate the number of composite multiplyaccumulate operations for a single image. Note that the composite multipleaccumulate operations are calculated for the image size reported in the table. Model size for [25] calculated from opensource implementation. Model
# parameters
MultAdds
Top 1 Acc. (%)
Top 5 Acc. (%)
Inception V1 [58] MobileNet224 [24] ShuffleNet (2x) [69]
6.6M 4.2 M ∼ 5M
1,448 M 569 M 524 M
69.8 70.6 70.9
89.9 89.5 89.8
NASNetA (4 @ 1056) NASNetB (4 @ 1536) NASNetC (3 @ 960)
5.3 M 5.3M 4.9M
564 M 488 M 558 M
74.0 72.8 72.5
91.6 91.3 91.0
Table 3. Performance on ImageNet classification on a subset of models operating in a constrained computational setting, i.e., < 1.5 B multiplyaccumulate operations per image. All models use 224x224 images.
sults obtaining 70.6% and 70.9% accuracy, respectively on 224x224 images using ∼550M multliplyadd operations. An architecture constructed from the best convolutional cells achieves superior predictive performance (74.0% ac
curacy) surpassing previous models but with comparable computational demand. In summary, we find that the learned convolutional cells are flexible across model scales achieving stateoftheart performance across almost 2 or
ders of magnitude in computational budget.
3.4. Efficiency of architecture search methods
3.3. Improved features for object detection
2 A primary advance in the best reported object detection system is the introduction of a novel loss [36]. Pairing this loss with NASNetA image featurization may lead to even further performance gains. Additionally, performance gains are achievable through ensembling multiple inferences across multiple model instances and image crops (e.g., [28]).
0.930 0.925 Accuracy at 20 Epochs
Image classification networks provide generic image features that may be transferred to other computer vision problems [13]. One of the most important problems is the spatial localization of objects within an image. To further validate the performance of the family of NASNetA networks, we test whether object detection systems derived from NASNetA lead to improvements in object detection [28]. To address this question, we plug in the family of NASNetA networks pretrained on ImageNet into the FasterRCNN object detection pipeline [46] using an opensource software platform [28]. We retrain the resulting object detection pipeline on the combined COCO training plus validation dataset excluding 8,000 minivalidation images. We perform single model evaluation using 300500 RPN proposals per image. In other words, we only pass a single image through a single network. We evaluate the model on the COCO minival [28] and testdev dataset and report the mean average precision (mAP) as computed with the standard COCO metric library [37]. We perform a simple search over learning rate schedules to identify the best possible model. Finally, we examine the behavior of two object detection systems employing the best performing NASNetA image featurization (NASNetA, 6 @ 4032) as well as the image featurization geared towards mobile platforms (NASNetA, 4 @ 1056). For the mobileoptimized network, our resulting system achieves a mAP of 29.6% – exceeding previous mobileoptimized networks that employ FasterRCNN by over 5.0% (Table 4). For the best NASNet network, our resulting network operating on images of the same spatial resolution (800 × 800) achieves mAP = 40.7%, exceeding equivalent object detection systems based off lesser performing image featurization (i.e. InceptionResNetv2) by 4.0% [28, 51] (see Appendix for example detections on images and sidebyside comparisons). Finally, increasing the spatial resolution of the input image results in the best reported, single model result for object detection of 43.1%, surpassing the best previous best by over 4.0% [36].2 These results provide further evidence that NASNet provides superior, generic image features that may be transferred across other computer vision tasks. Figure 9 and Figure 10 in Appendix C show four examples of object detection results produced by NASNetA with the FasterRCNN framework.
0.920 0.915 0.910 0.905 0.900 0.895 0.890 0
RL Top 1 Unique Models RL Top 5 Unique Models RL Top 25 Unique Models RS Top 1 Unique Models RS Top 5 Unique Models RS Top 25 Unique Models
10000 20000 30000 40000 Number of Models Sampled
50000
Figure 6. Measuring the efficiency of random search (RS) to reinforcement learning (RL) for learning neural architectures. The xaxis measures the total number of model architectures sampled, and the yaxis is the validation performance after 20 epochs on a proxy CIFAR10 training task. We emphasize that the absolute performance on the proxy task is not important (see text) but only the relative gain from the initial state. Each pair of curves measures the mean accuracy across top ranking models identified by each algorithm.
An open question in this proposed method is the training efficiency of the architecture search algorithm. In this section, we demonstrate the effectiveness of reinforcement learning for architecture search on the CIFAR10 image classification problem and compare it to bruteforce random search (considered to be a very strong baseline for blackbox optimization [5]) given an equivalent amount of computational resources. We define the effectiveness of an architecture search algorithm as the increase in model performance from the initial architecture identified with the search method. Importantly, we emphasize that the absolute value of the model performance on the proxy task is less important as this artificially reflects irrelevant factors employed in the architecture search process, e.g., the number of training epochs and the specific model construction. Thus, we employ the increase in the model performance as a proxy for judging convergence of the architecture search algorithm. Figure 6 shows the performance of reinforcement learning (RL) and random search (RS) as more model architectures are sampled. Note that the best model identified with RL is significantly better than the best model found by RS by over 1% as measured by the proxy classification task on CIFAR10. Additionally, RL finds an entire range of models that are of superior quality to random search. We observe this in the mean performance of the top5 and top25
Model
resolution
mAP (minival)
mAP (testdev)
MobileNet224 [24] ShuffleNet (2x) [69] NASNetA (4 @ 1056)
600 × 600 600 × 600 600 × 600
19.8% 24.5%† 29.6%

ResNet101FPN [35] InceptionResNetv2 (GRMI) [28] InceptionResNetv2 (TDM) [51] NASNetA (6 @ 4032) NASNetA (6 @ 4032)
800 (short side) 600 × 600 600 × 1000 800 × 800 1200 × 1200
35.7% 37.3% 41.3% 43.2%
36.2% 35.6% 36.8% 40.7% 43.1%
ResNet101FPN (RetinaNet) [36]
800 (short side)

39.1%
Table 4. Object detection performance on COCO on minival and testdev datasets across a variety of image featurizations. All results are with the FasterRCNN object detection framework [46] from a single crop of an image. Top rows highlight mobileoptimized image featurizations, while bottom rows indicate computationally heavy image featurizations geared towards achieving best results. All minival results employ the same 8K subset of validation images in [28].
models identified in RL versus RS. We take these results to indicate that although RS may provide a viable search strategy, RL does significantly improve our ability to learn neural architectures.
4. Related Work The proposed method is related to previous work in hyperparameter optimization [43, 4, 5, 53, 54, 6, 39] – especially recent approaches in designing architectures such as Neural Fabrics [47], DiffRNN [40], MetaQNN [3] and DeepArchitect [42]. A more flexible class of methods for designing architecture is evolutionary algorithms [64, 16, 56, 30, 45, 41, 66], yet they have not had as much success at large scale. Xie and Yuille [66] also transferred learned architectures from CIFAR10 to ImageNet but performance of these models (top1 accuracy 72.1%) are notably below previous stateoftheart (Table 2). The concept of having one neural network interact with a second neural network to aid the learning process, or learning to learn or metalearning [23, 48] has attracted much attention in recent years [1, 61, 14, 19, 34, 44, 15]. Most of these approaches have not been scaled to large problems like ImageNet. An exception is the recent work focused on learning an optimizer for ImageNet classification that achieved notable improvements [63]. The design of our search space took much inspiration from LSTMs [22], and Neural Architecture Search Cell [70]. The modular structure of the convolutional cell is also related to previous methods on ImageNet such as VGG [52], Inception [58, 59, 57], ResNet/ResNext [20, 67], and Xception/MobileNet [9, 24].
5. Conclusion In this work, we demonstrate how to learn scalable, convolutional cells from data that transfer to multiple image
classification tasks. The learned architecture is quite flexible as it may be scaled in terms of computational cost and parameters to easily address a variety of problems. In all cases, the accuracy of the resulting model exceeds all humandesigned models – ranging from models designed for mobile applications to computationallyheavy models designed to achieve the most accurate results. The key insight in our approach is to design a search space that decouples the complexity of an architecture from the depth of a network. This resulting search space permits identifying good architectures on a small dataset (i.e., CIFAR10) and transferring the learned architecture to image classifications across a range of data and computational scales. The resulting architectures approach or exceed stateoftheart performance in both CIFAR10 and ImageNet datasets with less computational demand than humandesigned architectures [59, 29, 68]. The ImageNet results are particularly important because many stateoftheart computer vision problems (e.g., object detection [28], face detection [49], image localization [62]) derive image features or architectures from ImageNet classification models. For instance, we find that image features obtained from ImageNet used in combination with the FasterRCNN framework achieves stateoftheart object detection results. Finally, we demonstrate that we can use the resulting learned architecture to perform ImageNet classification with reduced computational budgets that outperform streamlined architectures targeted to mobile and embedded platforms [24, 69].
References [1] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981–3989, 2016.
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [3] B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. In International Conference on Learning Representations, 2016. [4] J. Bergstra, R. Bardenet, Y. Bengio, and B. K´egl. Algorithms for hyperparameter optimization. In Neural Information Processing Systems, 2011. [5] J. Bergstra and Y. Bengio. Random search for hyperparameter optimization. Journal of Machine Learning Research, 2012. [6] J. Bergstra, D. Yamins, and D. D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International Conference on Machine Learning, 2013. [7] J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous sgd. In International Conference on Learning Representations Workshop Track, 2016. [8] Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. Dual path networks. arXiv preprint arXiv:1707.01083, 2017. [9] F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. [10] D.A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representations, 2016. [11] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. FeiFei. Imagenet: A largescale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009. [12] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. [13] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning, volume 32, pages 647–655, 2014. [14] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RL2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016. [15] C. Finn, P. Abbeel, and S. Levine. Modelagnostic metalearning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. [16] D. Floreano, P. D¨urr, and C. Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 2008. [17] K. Fukushima. A selforganizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, page 93202, 1980. [18] X. Gastaldi. Shakeshake regularization of 3branch residual networks. In International Conference on Learning Representations Workshop Track, 2017. [19] D. Ha, A. Dai, and Q. V. Le. Hypernetworks. In International Conference on Learning Representations, 2017.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. [21] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016. [22] S. Hochreiter and J. Schmidhuber. Long shortterm memory. Neural Computation, 1997. [23] S. Hochreiter, A. Younger, and P. Conwell. Learning to learn using gradient descent. Artificial Neural Networks, pages 87–94, 2001. [24] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. [25] J. Hu, L. Shen, and G. Sun. Squeezeandexcitation networks. arXiv preprint arXiv:1709.01507, 2017. [26] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. [27] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. [28] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al. Speed/accuracy tradeoffs for modern convolutional object detectors. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. [29] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations, 2015. [30] R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Learning Representations, 2015. [31] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. [32] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. [33] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 1998. [34] K. Li and J. Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017. [35] T.Y. Lin, P. Doll´ar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. [36] T.Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll´ar. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. [37] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014.
[38] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. [39] H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter. Towards automaticallytuned neural networks. In Proceedings of the 2016 Workshop on Automatic Machine Learning, pages 58–65, 2016. [40] T. Miconi. Neural networks with differentiable structure. arXiv preprint arXiv:1606.06216, 2016. [41] R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, A. Navruzyan, N. Duffy, and B. Hodjat. Evolving deep neural networks. arXiv preprint arXiv:1703.00548, 2017. [42] R. Negrinho and G. Gordon. DeepArchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792, 2017. [43] N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox. A highthroughput screening approach to discovering good forms of biologically inspired visual representation. PLoS Computational Biology, 5(11):e1000579, 2009. [44] S. Ravi and H. Larochelle. Optimization as a model for fewshot learning. In International Conference on Learning Representations, 2017. [45] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, Q. Le, and A. Kurakin. Largescale evolution of image classifiers. In International Conference on Machine Learning, 2017. [46] S. Ren, K. He, R. Girshick, and J. Sun. Faster RCNN: Towards realtime object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91–99, 2015. [47] S. Saxena and J. Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems, 2016. [48] T. Schaul and J. Schmidhuber. Metalearning. Scholarpedia, 2010. [49] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823, 2015. [50] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [51] A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Topdown modulation for object detection. arXiv preprint arXiv:1612.06851, 2016. [52] K. Simonyan and A. Zisserman. Very deep convolutional networks for largescale image recognition. In International Conference on Learning Representations, 2015. [53] J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Neural Information Processing Systems, 2012. [54] J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Ali, R. P. Adams, et al. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, 2015. [55] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neu
[56]
[57]
[58]
[59]
[60]
[61]
[62]
[63]
[64]
[65]
[66] [67]
[68]
[69]
[70]
ral networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. K. O. Stanley, D. B. D’Ambrosio, and J. Gauci. A hypercubebased encoding for evolving largescale neural networks. Artificial Life, 2009. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi. Inceptionv4, InceptionResnet and the impact of residual connections on learning. In International Conference on Learning Representations Workshop Track, 2016. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. J. X. Wang, Z. KurthNelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016. T. Weyand, I. Kostrikov, and J. Philbin. Planetphoto geolocation with convolutional neural networks. In European Conference on Computer Vision, 2016. O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. de Freitas, and J. SohlDickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813, 2017. D. Wierstra, F. J. Gomez, and J. Schmidhuber. Modeling systems with internal state using evolino. In The Genetic and Evolutionary Computation Conference, 2005. R. J. Williams. Simple statistical gradientfollowing algorithms for connectionist reinforcement learning. In Machine Learning, 1992. L. Xie and A. Yuille. Genetic CNN. arXiv preprint arXiv:1703.01513, 2017. S. Xie, R. Girshick, P. Doll´ar, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. X. Zhang, Z. Li, C. C. Loy, and D. Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. X. Zhang, X. Zhou, L. Mengxiao, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083, 2017. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
Appendix A. Experimental Details A.1. Dataset for Architecture Search The CIFAR10 dataset [31] consists of 60,000 32x32 RGB images across 10 classes (50,000 train and 10,000 test images). We partition a random subset of 5,000 images from the training set to use as a validation set for the controller RNN. All images are whitened and then undergone several data augmentation steps: we randomly crop 32x32 patches from upsampled images of size 40x40 and apply random horizontal flips. This data augmentation procedure is common among related work.
A.2. Controller architecture The controller RNN is a onelayer LSTM [22] with 100 hidden units at each layer and 2 × 5B softmax predictions for the two convolutional cells (where B is typically 5) associated with each architecture decision. Each of the 10B predictions of the controller RNN is associated with a probability. The joint probability of a child network is the product of all probabilities at these 10B softmaxes. This joint probability is used to compute the gradient for the controller RNN. The gradient is scaled by the validation accuracy of the child network to update the controller RNN such that the controller assigns low probabilities for bad child networks and high probabilities for good child networks. Unlike [70], who used the REINFORCE rule [65] to update the controller, we employ Proximal Policy Optimization (PPO) [50] with learning rate 0.00035 because training with PPO is faster and more stable. To encourage exploration we also use an entropy penalty with a weight of 0.00001. In our implementation, the baseline function is an exponential moving average of previous rewards with a weight of 0.95. The weights of the controller are initialized uniformly between 0.1 and 0.1.
A.3. Training of the Controller For distributed training, we use a workqueue system where all the samples generated from the controller RNN are added to a global workqueue. A free “child” worker in a distributed worker pool asks the controller for new work from the global workqueue. Once the training of the child network is complete, the accuracy on a heldout validation set is computed and reported to the controller RNN. In our experiments we use a child worker pool size of 450, which means there are 450 networks being trained on 450 GPUs concurrently at any time. Upon receiving enough child model training results, the controller RNN will perform a gradient update on its weights using PPO and then sample another batch of architectures that go into the global workqueue. This process continues until a predetermined
number of architectures have been sampled. In our experiments, this predetermined number of architectures is 20,000 which means the search process is terminated after 20,000 child models have been trained. Additionally, we update the controller RNN with minibatches of 20 architectures. Once the search is over, the top 250 architectures are then chosen to train until convergence on CIFAR10 to determine the very best architecture.
A.4. Details of architecture search space We performed preliminary experiments to identify a flexible, expressive search space for neural architectures that learn effectively. Generally, our strategy for preliminary experiments involved smallscale explorations to identify how to run largescale architecture search. • All convolutions employ ReLU nonlinearity. Experiments with ELU nonlinearity [10] showed minimal benefit. • To ensure that the shapes always match in convolutional cells, 1x1 convolutions are inserted as necessary. • Unlike [24], all depthwise separable convolution do not employ Batch Normalization and/or a ReLU between the depthwise and pointwise operations. • All convolutions followed an ordering of ReLU, convolution operation and Batch Normalization following [21]. • Whenever a separable convolution is selected as an operation by the model architecture, the separable convolution is applied twice to the hidden state. We found this empirically to improve overall performance.
A.5. Training with stochastic regularization We performed several experiments with various stochastic regularization methods. Naively applying dropout [55] across convolutional filters degraded performance. However, when training NASNet models, we found that stochastically dropping out each path (i.e., edge with a yellow box in Figure 4) in the cell with some fixed probability to be an effective regularizer. This is similar to [27] and [68] where they dropout full parts of their model during training and then at test time scale the path by the probability of keeping that path during training. Interestingly we found that linearly increasing the probability of dropping out a path over the course of training to significantly improve the final performance for both CIFAR and ImageNet experiments.
A.6. Training of CIFAR models All of our CIFAR models use a single period cosine decay as in [38, 18]. All models use the momentum optimizer
with momentum rate set to 0.9. All models also use L2 weight decay. Each architecture is trained for a fixed 20 epochs on CIFAR10 during the architecture search process. Additionally, we found it beneficial to use the cosine learning rate decay during the 20 epochs the CIFAR models were trained as this helped to further differentiate good architectures. We also found that having the CIFAR models use a small N = 2 during the architecture search process allowed for models to train quite quickly, while still finding cells that work well once more were stacked.
A.7. Training of ImageNet models We use ImageNet 2012 ILSVRC challenge data for large scale image classification. The dataset consists of ∼ 1.2M images labeled across 1,000 classes [11]. Overall our training and testing procedures are almost identical to [59]. ImageNet models are trained and evaluated on 299x299 or 331x331 images using the same data augmentation procedures as described previously [59]. We use distributed synchronous SGD to train the ImageNet model with 50 workers (and 3 backup workers) each with a Tesla K40 GPU [7]. We use RMSProp with a decay of 0.9 and epsilon of 1.0. Evaluations are calculated using with a running average of parameters over time with a decay rate of 0.9999. We use label smoothing with a value of 0.1 for all ImageNet models as done in [59]. Additionally, all models use an auxiliary classifier located at 2/3 of the way up the network. The loss of the auxiliary classifier is weighted by 0.4 as done in [59]. We empirically found our network to be insensitive to the number of parameters associated with this auxiliary classifier along with the weight associated with the loss. All models also use L2 regularization. The learning rate decay scheme is the exponential decay scheme used in [59]. Dropout is applied to the final softmax matrix with probability 0.5.
B. Additional Experiments We now present two additional cells that performed well on CIFAR and ImageNet. The search spaces used for these cells are slightly different than what was used for NASNetA. For the NASNetB model in Figure 7 we do not concatenate all of the unused hidden states generated in the convolutional cell. Instead all of the hiddenstates created within the convolutional cell, even if they are currently used, are fed into the next layer. Note that B = 4 and there are 4 hiddenstates as input to the cell as these numbers must match for this cell to be valid. We also allow addition followed by layer normalization [2] or instance normalization [60] to be predicted as two of the combination operations within the cell, along with addition or concatenation. For NASNetC (Figure 8), we concatenate all of the unused hidden states generated in the convolutional cell like in NASNetA, but now we allow the prediction of addition
Figure 7. Architecture of NASNetB convolutional cell with B = 4 blocks identified with CIFAR10. The input (white) is the hidden state from previous activations (or input image). Each convolutional cell is the result of B blocks. A single block is corresponds to two primitive operations (yellow) and a combination operation (green). As do we not concatenate the output hidden states, each output hidden state is used as a hidden state in the future layers. Each cell takes in 4 hidden states and thus needs to also create 4 output hidden states. Each output hidden state is therefore labeled with 0, 1, 2, 3 to represent the next four layers in that order.
followed by layer normalization or instance normalization like in NASNetB.
C. Example object detection results Finally, we will present examples of object detection results on the COCO dataset in Figure 9 and Figure 10. As can be seen from the figures, NASNetA featurization works well with FasterRCNN and gives accurate localization of objects.
Figure 9. Example detections showing improvements of object detection over previous stateoftheart model for FasterRCNN with InceptionResNetv2 featurization [28] (top) and NASNetA featurization (bottom).
Figure 8. Architecture of NASNetC convolutional cell with B = 4 blocks identified with CIFAR10. The input (white) is the hidden state from previous activations (or input image). The output (pink) is the result of a concatenation operation across all resulting branches. Each convolutional cell is the result of B blocks. A single block corresponds to two primitive operations (yellow) and a combination operation (green).
Figure 10. Example detections of best performing NASNetA featurization with FasterRCNN trained on COCO dataset. Top and middle images courtesy of http://wikipedia.org. Bottom image courtesy of Jonathan Huang