@hoaxuan7074

ReLU is a switch. f(x)=x is connect, f(x)=0 is disconnect. A light switch in your house is binary on off,  yet connects and disconnects a continuously variable AC voltage signal.  Then a ReLU net is a switched composition of weighted sums. When all the switch states become known the connected weighted sums can be simplified down. The net collapses to a simple matrix mapping the input vector to the output vector. You can examine the matrix with various metrics. Eg. Noise sensitivity. 
Some other net components like convolution and max pooling can be viewed as weighted sums and switching too.

@hoaxuan7074

As you store more <vector,scalar> associations using a dot product with a weight vector the larger the angles to the weight vector and the larger the length of the weight vector. Hence one attack is to input a vector pointing in the same direction as the weight vector to get an extreme response. Similar to supernormal stimuli in biological neural networks       
AI462 neural networks.