tf.lin_space(start, stop, num, name=None)

create a sequence of num evenly-spaced values are generated beginning at start. If num > 1, the values in the sequence increase by (stop - start) / (num - 1), so that the last one is exactly stop.
comparable to but slightly different from numpy.linspace

lin = tf.lin_space(10.0, 13.0, 4, name="linspace")
with tf.Session() as sess:
    print(sess.run(lin))

output:

[10. 11. 12. 13.]

tf.range([start], limit=None, delta=1, dtype=None, name='range')

create a sequence of numbers that begins at start and extends by increments of delta up to but not including limit

slight different from range in Python

# 'start' is 3, 'limit' is 18, 'delta' is 3
tf.range(start, limit, delta) ==> [3, 6, 9, 12, 15]
# 'start' is 3, 'limit' is 1,  'delta' is -0.5
tf.range(start, limit, delta) ==> [3, 2.5, 2, 1.5]
# 'limit' is 5
tf.range(limit) ==> [0, 1, 2, 3, 4]

Unlike Numpy or other Python sequences, TensorFlow sequences cannot iterate

for _ in np.linspace(0, 10, 4): # OK
for _ in tf.linspace(0.0, 10.0, 4): # TypeError: 'Tensor' object is not iterable.

for _ in range(4): # OK
for _ in tf.range(4): # TypeError: 'Tensor' object is not iterable.

Create variables

To declare a variable, you need to instantiate a tf.Variable, paying attention to V capitalization.

x = tf.Variable(...) 
x.initializer # init 
x.value() # read op 
x.assign(...) # write op 
x.assign_add(...) 
# and more

The old way to create variables is to call tf.Variable(, name=)

s = tf.Variable(2, name="scalar") 
m = tf.Variable([[0, 1], [2, 3]], name="matrix") 
W = tf.Variable(tf.zeros([784,10]))

However, this approach has been abandoned by TensorFlow and we recommend that we use tf.get_variable to create it, because it can better achieve variable sharing. With tf.get_variable, we can provide the initial values with internal name, shape, type and initializer. Note that when we use tf.constant as initializer, we don't need to provide shape.

tf.get_variable(
    name,
    shape=None,
    dtype=None,
    initializer=None,
    regularizer=None,
    trainable=True,
    collections=None,
    caching_device=None,
    partitioner=None,
    validate_shape=True,
    use_resource=None,
    custom_getter=None,
    constraint=None
)

s = tf.get_variable("scalar", initializer=tf.constant(2)) 
m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))
W = tf.get_variable("big_matrix", shape=(784, 10), initializer=tf.zeros_initializer())

initialize variable

Before using a variable, you need to initialize it, otherwise you will report an error: Failed PreconditionError: Attempting to use uninitialized value. You can print the uninitialized variable by the following statement:

print(tf.Session().run(tf.report_uninitialized_variables()))

(1) Initialize all variables: tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

(2) Initialize part of the variable tf.variables_initializer()

with tf.Session() as sess:
    sess.run(tf.variables_initializer([a, b]))

(3) Initialize a single variable tf.Variable.initializer

with tf.Session() as sess:
    sess.run(W.initializer)

View the value of a variable

Remove the value from session

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run([s, m, W]))

output:

[2, array([[0, 1],
       [2, 3]], dtype=int32), array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)]

Get the value through tf.Variable.eval()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(s.eval(), m.eval())

output:

2 [[0 1]
 [2 3]]

Assignment of variables

By tf.Variable.assign()

W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
    sess.run(W.initializer)
    print(W.eval()) # >> 10

Why output 10 instead of 100? ** W.assign(100) does not assign 100 to W, but creates an assign op. ** To make this op work, we need to run op in session.

W = tf.Variable(10)

assign_op = W.assign(100)
with tf.Session() as sess:
    sess.run(assign_op)
    print(W.eval()) # >> 100

Note that we don't have to initialize W at this point, because assign() has been implemented for us.

# create a variable whose original value is 2
a = tf.get_variable('scalar', initializer=tf.constant(2)) 
a_times_two = a.assign(a * 2)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer()) 
    sess.run(a_times_two) # >> 4
    sess.run(a_times_two) # >> 8
    sess.run(a_times_two) # >> 16

tf.Variable.assign_add() and tf.Variable.assign_sub(), which need to be initialized

W = tf.Variable(10)

with tf.Session() as sess:
    sess.run(W.initializer)
    print(sess.run(W.assign_add(10))) # >> 20
    print(sess.run(W.assign_sub(2)))  # >> 18

Variables between session s are independent of each other

W = tf.Variable(10)
sess1 = tf.Session()
sess2 = tf.Session()
sess1.run(W.initializer)
sess2.run(W.initializer)
print(sess1.run(W.assign_add(10)))      # >> 20
print(sess2.run(W.assign_sub(2)))       # >> 8
print(sess1.run(W.assign_add(100)))     # >> 120
print(sess2.run(W.assign_sub(50)))      # >> -42
sess1.close()
sess2.close()

Import data

Placement holder and feed_dict (old method)

TensorFlow program execution usually consists of two stages

1: Declare a graph
 2: Use a session to perform calculations to evaluate variables in the graph

When you declare graphs, you don't need to know the value of the variable you want to calculate. It's like a real-name function about X and y: f(x,y)=2x+y, where x and y are placeholders for the actual value.

After the graph is created, we need to define a placeholder when we provide the value later:
tf.placeholder(dtype, shape=None, name=None)

Note that dtype and shape need to be declared by themselves; when shape=none, tensors that can accept arbitrary shapes are indicated.

a = tf.placeholder(tf.float32, shape=[3]) # a is placeholder for a vector of 3 elements
b = tf.constant([5, 5, 5], tf.float32)
c = a + b # use the placeholder as you would any tensor
with tf.Session() as sess:
    print(sess.run(c)) 

At this point, the execution of the above program will report an error, because it has not been provided with a value.
Now pass the value to a using feed_dict

with tf.Session() as sess:
    # compute the value of c given the value of a is [1, 2, 3]
    print(sess.run(c, feed_dict={a: [1, 2, 3]}))        # [6. 7. 8.]

Tf. data (new method)

Comparing with tf.placeholder and feed_dict, it improves the operation performance.

Batch standardization

batch normalization (BN) is generally used before the activation function, so that the result x=Wx+b has a mean value of 0 and a variance of 1. By normalizing the distribution of activation functions in linear intervals, a stable distribution of input at each level will be beneficial to the training of the network.
Advantage:

• Increase the exploration step and speed up the convergence rate.
• It is easier to jump out of local minima.
• The original data distribution is destroyed to prevent over-fitting to a certain extent.
• Solve slow convergence speed and gradient explosion.

tensorflow API

mean, variance = tf.nn.moments(x, axes, name=None, keep_dims=False) 

To compute statistical moments, mean is the first moment, mean, variance is the second central moment, variance, axes=[0] is calculated by column.

tf.nn.batch_normalization(x, mean, variance, offset, scale, variance_epsilon, name=None) 
tf.nn.batch_norm_with_global_normalization(x, mean, variance, beta, gamma, variance_epsilon, scale_after_normalization, name=None)；

tf.nn.moments computes the returned mean and variance as tf.nn.batch_normalization parameter calls

tensorflow and python implementation

import tensorflow as tf
W = tf.constant([[-2.,12.,6.],[3.,2.,8.]], )
mean,var = tf.nn.moments(W, axes = [0])

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())  # #You have to add this sentence or you'll make a mistake if you execute sess several times.
    resultMean = sess.run(mean)
    print(resultMean)
    resultVar = sess.run(var)
    print(resultVar)

[ 0.5  7.   7. ]
[  6.25  25.     1.  ]

tf.nn.embedding_lookup()

The main purpose is to select the corresponding elements of the index in a tensor.
tf.nn.embedding_lookup(params, ids):
params can be tensors, arrays, etc.
id is the corresponding index
Other parameters are not introduced.
ids has only one line:

#c = np.random.random([10, 1])  # Random generation of an array of 10*1
#b = tf.nn.embedding_lookup(c, [1, 3])#Find arrays with numbers 1 and 3
p=tf.Variable(tf.random_normal([10,1]))#Generating tensors of 10*1
b = tf.nn.embedding_lookup(p, [1, 3])#Find tensors with numbers 1 and 3
 
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(b))
    #print(c)
    print(sess.run(p))
    print(p)
    print(type(p))

[[0.15791859]
 [0.6468804 ]]
[[-0.2737084 ]
 [ 0.15791859]
 [-0.01315552]
 [ 0.6468804 ]
 [-1.4090979 ]
 [ 2.1583703 ]
 [ 1.4137447 ]
 [ 0.20688428]
 [-0.32815856]
 [-1.0601649 ]]
<tf.Variable 'Variable:0' shape=(10, 1) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>

If ids is multi-line:

import tensorflow as tf
import numpy as np

a = [[0.1, 0.2, 0.3], [1.1, 1.2, 1.3], [2.1, 2.2, 2.3], [3.1, 3.2, 3.3], [4.1, 4.2, 4.3]]
a = np.asarray(a)
idx1 = tf.Variable([0, 2, 3, 1], tf.int32)
idx2 = tf.Variable([[0, 2, 3, 1], [4, 0, 2, 2]], tf.int32)
out1 = tf.nn.embedding_lookup(a, idx1)
out2 = tf.nn.embedding_lookup(a, idx2)
init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    print (sess.run(out1))
    print(out1)
    print ('==================')
    print(sess.run(out2))
    print(out2)

[[ 0.1  0.2  0.3]
 [ 2.1  2.2  2.3]
 [ 3.1  3.2  3.3]
 [ 1.1  1.2  1.3]]
Tensor("embedding_lookup:0", shape=(4, 3), dtype=float64)
==================
[[[ 0.1  0.2  0.3]
  [ 2.1  2.2  2.3]
  [ 3.1  3.2  3.3]
  [ 1.1  1.2  1.3]]

 [[ 4.1  4.2  4.3]
  [ 0.1  0.2  0.3]
  [ 2.1  2.2  2.3]
  [ 2.1  2.2  2.3]]]
Tensor("embedding_lookup_1:0", shape=(2, 4, 3), dtype=float64)

Reference resources: https://www.cnblogs.com/gaofighting/p/9625868.html
https://www.jianshu.com/p/2a822b0ce042