Neural Architecture Optimization, NIPS

Abstract

Propose a simple and efficient method to automatic neural architecture design based on continuous optimization

3 components:

• an encoder embeds/maps neural network architectures into a continuous space
• a predictor takes the continuous representation of a network as input and predicts its accuracy
• a decoder maps a continuous representation of a network back to its architecture

competitive for CIFAR-10 and PTB

successfully transfered

limited computational resources, less than 10 GPU hours

Designing neural network architectures using Reinforcement, ICLR 2017

Abstract

MetaQNN, a meta-modeling algorithm based on reinforcement learning

Learning agent is trained to sequentially choose CNN layers using Q-learning with an $\epsilon$-greedy exploration strategy and experience replay

Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Abstract

Bayesian Optimisation refers to a class of methods for global optimisation of a function $f$ which is only accessible via point evaluations

Typically used in setting where $f$ is expensive to evaluate

Common use case for BO in ml is model selection

Conventional BO methods have focused on Euclidean and categorical domains, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms

NASBOT, a Gaussian process based on BO framework for NAS

Develop a distance metric in the space of neural network architecture which can be computed efficiently via an optimal transport program

Large-Scale Evolution of Image Classifiers

Abstract

Use novel and intuitive mutation operators that navigate large search spaces

Stress that no human participation is required once evolution starts and that the output is a fully-trained model

Repeatability of results, variability in the outcomes, computational requirements

ICLR 2019

Abstract

In a differential manner

Based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient search

Being orders of magnitude faster than state-of-the-art non-differentiable techniques

Implement publicly available

Random Search and Reproducibility for Neural Architecture Search

Abstract

Two points:

1. Evaluate both random search with early-stopping and a novel random search with weight-shared algorithm. Results show that random search with early-stopping is a competitive NAS baseline.
2. Explore the existing reproducibility issues of published NAS results.

AutoML

paper：AutoML综述

Taking Human out of learning application: a survey on automated machine learning

• Feature engineering
• model selection
• optimization algorithm selection

Optimizer & Controller

• For Optimizer：
  • Simple search approaches: Greedy search, random search
  • Optimization from samples
    • Heuristic search
    • Model based Derivative-Free Optimization
    • RL
  • Gradient descent
  • Greedy search
• For evaluator
  • Direct evaluator
  • sub-sampling
  • early stop
  • parameter reusing
  • surrogate evaluator
• Experienced techniques
  • Meta Learning
  • Transfer Learning

Efficient Neural Architecture Search via Parameter Sharing

Abstract

ENAS, a controller discovers neural network architectures by searching for an optimal subgraph within a large computation graph

PG

sharing parameters among child models

much fewer GPU-hours

Tensor

A tensor consists of a set of primitive values shaped into array of any number of dimensions.

Tensorflow Core WalkThrough

1. Building the computational graph – tf.Graph
2. Running the computational graph – tf.Session

Graph

contains two types of objects

• Operations: the node of the graph
• Tensors: the edges in the graph

Session

1
2

sess = tf.Session()
sess.run()

Feeding

1
2
3
4
5

x = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
z = x + y
print(sess.run(z, feed_dict={x: 3, y: 4.5}))
print(sess.run(z, feed_dict={x: [1, 3], y: [2, 4]}))

[TOC]

Neural Architecture Search with Reinforcement Learning

Abstract

Use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set

Introduction

A gradient-based method for finding architecture

Work is based on the observation that the structure and connectivity of a neural network can be typically specified by a variable-length string. Use a recurrent network, the controller, to generate such string

Reward: result in an accuracy on a validation set

Controller will give higher probabilities to architectures that receive high accuracies