Disadvantages of static graph

The original version of tensorflow was run in the way of static graph. In this way, computation graph separated the definition and execution of computation. This is a declarative programming model

The execution mode of static graph has many advantages, but it's really inconvenient when debugging (similar to calling compiled C program, at this time we can't debug it internally). Therefore, with Eager Execution, it was first introduced in TensorFlow v1.5 and became the core API in version 2.0.

After the introduction of Eager Execution mode, TensorFlow has the same dynamic graph model capability as python. We don't need to wait for see.run(*) to see the execution results. It is convenient to debug the code at any time in IDE and view the OPs execution results. The introduction of dynamic graph also brings some new features to write tf code, which should be noted.

Eager mode

The Eager mode is a bit similar to python's command-based programming. It does not need to be compiled and run directly, which is very intuitive.

Basic features of Eagle execution

Support for numpy

The support for numpy in eager mode is very friendly. The specific features are as follows:

• The operation of numpy can take Tensor as parameter;
• tensorflow's mathematical operation will convert python objects and numpy's arrays into Tensor;
• tf.Tensor.numpy method returns the ndarray of numpy

For example:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Jan  9 10:21:24 2020

@author: lxy_alex@outlook.com
"""

import numpy as np
import tensorflow as tf
tf.compat.v1.enable_eager_execution()

def example_of_tf_and_np():

    a = tf.constant([[1,2],[3,4]])
    b = tf.add(a,1)
    
    print(a)
    print(b)
    
    print('tf\'s multiply: ')
    print(a*b)
    
    c = np.multiply(a,b)
    print('numpy\'s multiply:')
    print(c)
    
    print('transfer tensor a to numpy ndarray from: ')
    print(a.numpy())
    
if __name__ == '__main__':
    example_of_tf_and_np()

Get:

tf's multiply: 
tf.Tensor(
[[ 2  6]
 [12 20]], shape=(2, 2), dtype=int32)
numpy's multiply:
[[ 2  6]
 [12 20]]
transfer tensor a to numpy ndarray from: 
[[1 2]
 [3 4]]

Although tensorflow's eager mode has good compatibility with the multidimensional data of tensor and numpy, it does not mean that the variables defined by tf.Tensor() are the same as other variables of Python. In practical use, it must be noted that the python variable and tf's tensor object cannot be confused.

Auto Graph - dynamic graph

eager mode supports both python control flow and tf dynamic flow. For tf dynamic flow, for while loop or similar loop (for, if control may be used), the form is as follows:

while x>0:
    x = x-1

In tensorflow control flow, it can be written as TF. While , loop? Vars = (x,)). However, tf.while-loop can not support infinite variables. Meanwhile, the efficiency of tensorflow calculation graph is affected by the number of while loop loops in it, so it can not be used at will.

AutoGraph uses static analysis to determine which symbols have been modified by the code in order to convert them to control rheology. Static analysis is usually performed on a single function -- Python's dynamic nature limits its effectiveness across functions.

static analysis VS dynamic flow

Visible domain of local parameter

After the local variables in the function change, the change is invisible in the main program outside the function. Similarly, in the method defined by the class, when the local variables change, the main program is invisible, unless these variables are explicitly returned as output parameters. Similarly, parameters inside a class member function are invisible outside the function.

The use of python collections data in tensorflow control flow

The control flow of tf supports most Python data structures, such as lists, dictionaries and tuples, including the namedtuple object of the collection object. However, in the control flow of tf, these variables are allowed to be fixed structure, that is, in the loop, the list cannot change the length, and the dictionary cannot increase or decrease the keys. What is a namedtuple, please refer to: https://docs.python.org/3/library/collections.htmlාcollections.namedtuple

def fn():
  l = []

  def loop_cond(i):
    return i < 10

  def loop_body(i):
    i = i + 1
    l.append(i)
    return i,

  tf.while_loop(
      cond=loop_cond,
      body=loop_body,
      loop_vars=(0,))

  return l

print(fn()) # Output: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

tf.function(fn)() # ERROR

The code that can be run under eager execution reports an error under tf. Function (fn()). This is because tf.function() will start graph execution, and tf uses a special mechanism for graph execution to ensure the correctness of the operation sequence.

Another example is as follows:

def fnn():
    l = []
    for i in tf.range(10):
      l.append(i)  # Error -- illegal tensor capture!
    return l

Directly execute ll=fnn() in eager execution mode to get ll is a listener list of eager execution. However, tf.function(fnn)() is also used for execution, and the error is reported as follows:

InaccessibleTensorError: The tensor 'Tensor("placeholder:0", shape=(), dtype=int32)' cannot be accessed here: it is defined in another function or code block. Use return values, explicit Python locals or TensorFlow collections to access it. Defined in: FuncGraph(name=while_body_1396, id=5377892048); accessed from: FuncGraph(name=fnn, id=5374487632).

The correct way should be to define variables that l is tf.TensorArray(), and to invoke the write() method of TensorArray in the loop, and gradually increase the elements in TensorArray. When the local parameter L is defined, it is assigned with a length of 0 and a data type of int32, and the TensorArrary is set to be variable length (dynamic_size=True)

def fnn():
    l=tf.TensorArray(tf.int32,size=0,dynamic_size=True)
    for i in tf.range(10):
        l.write(l.size(), i)
    return l
tf.function(fnn)() 

Of course, the above fnn() function can also be directly executed in the eager execution mode (ll=fnn()), and the result is that ll is

ll
Out[188]: <tensorflow.python.ops.tensor_array_ops.TensorArray at 0x140957d90>

If python collections are included in tensorflow's process control, the index is variable, but the structure should be fixed.
For example:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat Jan 18 22:27:11 2020

@author: lxy_alex@outlook.com
"""

import tensorflow as tf


if tf.executing_eagerly():
    tf.compat.v1.disable_eager_execution()
    
@tf.function
def dict_loop():
    d = {'a': tf.constant(3)}
    for i in tf.range(10):
      d = {key: value + i for key, value in d.items()}
    return d

@tf.function
def dict_loop2():
    d = {'a': tf.constant(3)}
    for i in tf.range(10):
      for key in d:
        d[key] += i  # Problem -- accessing `dict` using non-constant key
    return d
''
d = dict_loop() # d={'a': <tf.Tensor 'StatefulPartitionedCall_3:0' shape=() dtype=int32>}
//however
d2 = dict_loop2() # ERROR

In this example, the way defined in the dict UU loop2() function is wrong, while the dict UU loop() function has no problem. The official explanation is that the functional style programming method should be used. When writing code, we must pay attention to this subtle difference.

Dimension and data type of sensor in tensor flow control flow

However, in tf's graph control flow, the sensor dimension and data type need to remain unchanged, but this limitation is not valid in eager Execuion mode, because in eager mode, python control flow is used. So when you change the code from eager mode to graph mode, you must pay attention to this problem.

Dynamic calculation and static dimension

The shape and rank of the sensor are defined as follows:

The static shape is obtained by the. Shape method, and the static rank of the sensor is obtained by the. shape.rank method. When the sensor is dinamic, its shape and rank should be obtained by tf.shape(), tf.rank(), respectively.

If dynamic dimensions are needed in the code, there are two methods:
1) You can use @ tf.function decorator, for example

@tf.function(input_signature=(tf.TensorSpec(shape=(None,))))
def f(x):  # x now has dynamic shape
  if tf.shape(x)[0] >= 3:  # Builds a tf.cond
    val = x[4]  # Okay, bounds checks are skipped when the shape is dynamic
  else:
    val = some_default_value

Here, after the input_signature is assigned, the tf will skip the shape related check when executing.
2) Using python to control the flow, add a check of whether the parameter is static or dynamic, such as

if x.shape[0] is None:  # Python bool, does not use tf.cond
  # ... use x.shape here ...
else:
  # ... use tf.shape(x) here ...

Consistency of dtype and shape

In tf process, it must be noted that dtype and shape should always be consistent, such as the following error code:

x = tf.cond(
    tf.random.uniform(()) > 0.5,
    lambda: tf.constant(1, dtype=tf.int32),
    lambda: tf.constant(1, dtype=tf.float32))  # Error -- inconsistent dtypes: int32, float32

# This won't work - "x" changes dtype inside the loop.
x = tf.while_loop(
    lambda _: tf.random.uniform(()) > 0.5,
    lambda x: tf.constant(1, dtype=tf.float32),
    loop_vars=(tf.constant(1, dtype=tf.int32),))  # Error -- inconsistent dtypes: int32, float32
# Example of illegal shape change in a loop:
x = tf.constant(1,)
while tf.random.uniform(()) > 0.5:
  x = tf.constant((1, 2, 3))  # Error -- inconsistent shapes: (), (3,)

If there are None or undefined conditions in the control flow, an error will also be reported.

Accessibility of source code

In eager mode, you can execute various original codes visible at runtime, but there are also exceptions:
1) Code cannot be executed in the python interactive environment, such as ipython or jupyter lab
2) Functions with native bindings, such as code in other languages
3) Dynamic code executed with exec or eval

Inspection. Getsource (object) can be used to check the code's reachability. https://docs.python.org/3/library/inspect.html#inspect.getsource

For functions of type lambda, for example:

foo = (
 'bar',
 lambda: x)

In this case, the definition of the function is in the lambda expression, and there is no problem. In case of nesting, the called function should be declared before calling, for example:

my_lambda = lambda: x
foo = ('bar', my_lambda)

Eager training mode

Let's start with this example:

w = tf.Variable([[1.0]])
# Forward calculation to get loss
with tf.GradientTape() as tape:
  loss = w * w

grad = tape.gradient(loss, w)
print(grad)  # => tf.Tensor([[ 2.]], shape=(1, 1), dtype=float32)

This is the training mode of eager execution. In eager mode, tf.GradientTape can be used to track and record. Tape can be visualized as a tape, so reverse calculation is equivalent to rewinding. Take multiple linear regression as an example:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat Jan 18 01:36:15 2020

@author: lxy_alex@outlook.com
"""
import tensorflow as tf


# A toy dataset of points around 3 * x + 2
# Add a little noise to generate training data;
NUM_EXAMPLES = 1000
training_inputs = tf.random.normal([NUM_EXAMPLES,4])
noise = tf.random.normal([NUM_EXAMPLES])

training_outputs = tf.matmul(training_inputs,[[2.7],[3.1],[5.4],[8.9]])+6.5+noise

def prediction(indata, weight, bias):
  return tf.matmul(indata, weight) + bias

# loss adopts mean square error
def loss(weights, biases):
  error = prediction(training_inputs, weights, biases) - training_outputs
  return tf.reduce_mean(tf.square(error))

# Return the derivative of loss with respect to weight and bias
def grad(weights, biases):
  # Forward calculation to get loss, and record the operation to tape for gradient calculation
  with tf.GradientTape() as tape:
    loss_value = loss(weights, biases)
  # Play the tape backward to get the gradient;
  return tape.gradient(loss_value, [weights, biases])

train_steps = 300
learning_rate = 0.01
# Start with arbitrary values for W and B on the same batch of data
W = tf.Variable([[0.],[0.],[0.],[0.]])
B = tf.Variable(0.)

print("Initial loss: {:.3f}".format(loss(W, B)))

for i in range(train_steps):
  dW, dB = grad(W, B)
  W.assign_sub(dW * learning_rate) # W = W - dW * learning_rate 
  B.assign_sub(dB * learning_rate) # B = B - dB * learning_rate
  if i % 50 == 0:
    print("Loss at step {:03d}: {:.3f}".format(i, loss(W, B)))

print("Final loss: {:.3f}".format(loss(W, B)))
print("W = {}, B = {}".format(W.numpy(), B.numpy()))

obtain

Initial loss: 161.488
Loss at step 000: 155.372
Loss at step 050: 23.209
Loss at step 100: 4.175
Loss at step 150: 1.404
Loss at step 200: 0.996
Loss at step 250: 0.936
Final loss: 0.927
W = [[2.6918666]
[3.0815856]
[5.377633 ]
[8.876133 ]], B = 6.478857517242432

Read more:

tf.Variable() and its assign ment

Operation interface of Variable: assign()

W = tf.Variable(10)
W.assign(100) 
with tf.Session() as sess: 
    sess.run(W.initializer)    
    print(W.eval(session=sess))

Is the result of printing 10 or 100???
The answer is: 10

This is because W.assign(100) does not assign a value to W. assign() is an op, so it returns an op object, which needs to be run in the Session to assign a value to W

W = tf.Variable(10)
assign_op = W.assign(100) 
with tf.Session() as sess:
     sess.run(W.initializer) 
     sess.run(assign_op) 
     print(W.eval())# >> 100

Underlined code can be omitted, because assign menu can complete initial value assignment. In fact, initializer op is a special assignment Op

https://cloud.tencent.com/developer/article/1082033

python collections

collections is a python built-in package, which is a very convenient data structure.
For details, please refer to Liao Xuefeng's python learning