Introduction to TensorFlow

A Quick Start

Yychi Fyu @SIST, ShanghaiTech

Outline

  1. TF Primitives
    • tensor
    • graphs
    • session
    • variable
    • placeholder
  2. An Example

High-level overview

Computation Graph: The structure of TF.

  • operators as nodes
  • tensors as links
computation graph
1
2
3
4
5
6
7

import tensorflow as tf

W = tf.constant([[1, 2]])
x = tf.constant([[3], [4]])
b = tf.constant(5)
y = tf.matmul(W, x) + b
print(y)

Tensor("add_5:0", shape=(1, 1), dtype=int32)

The above codes “describe” a computation graph:

How can we actually see a tensor?

To evaluate a tensor, we need

  1. an instance of session created by tf.Session()
  2. evaluate the tensor at the target session
1
2
3

sess = tf.Session()
y_value = y.eval(session=sess)
print(y_value)

[[16]]

This is so-called lazy-evaluation!

Keep in mind:

  1. TF code just “describes” computations and doesn’t actually perform it
  2. each node is a TF operator
  3. each link (edge) transports some tensors

tensors

tensor is most fundamental object in TF. Almost all TF operations return the reference of tensor.

  • rank-0 tensor = scalar
  • rank-1 tensor = vector
  • rank-2 tensor = matrix
Tensors

Create constant tensor

1
2
3
4
5
6
7
8
9

import tensorflow as tf
import numpy as np

a = tf.constant(2)          # rank-0
b = tf.constant([1, 2])     # rank-1
c = tf.constant([[1], [2]]) # rank-2
print(a)
print(b)
print(c)

Tensor("Const_15:0", shape=(), dtype=int32)
Tensor("Const_16:0", shape=(2,), dtype=int32)
Tensor("Const_17:0", shape=(2, 1), dtype=int32)

Mind the shape!

Interactive tensor evaluation

  1. Create zeros tensor
1
2

>>> tf.zeros(2)
<tf.Tensor 'zeros:0' shape=(2,) dtype=float32>
  1. Evaluate the value of a tensor
1
2
3
4
5

# use `tf.InteractiveSession()` for eager evaluation
>>> tf.InteractiveSession()
>>> a = tf.zeros(2)
>>> a.eval()
array([0., 0.], dtype=float32)

Tensor addition and scaling

1
2
3
4
5
6

>>> a = tf.ones(shape=(2, 2))
>>> b = tf.fill((2, 2), 2.0)
>>> c = a + b
>>> c.eval()
array([[3., 3.],
       [3., 3.]], dtype=float32)    

1
2
3
4

>>> d = 2 * c
>>> d.eval()
array([[6., 6.],
       [6., 6.]], dtype=float32)    

1
2
3
4
5
6
7

>>> e = c * d           # elem-wise multiplication
>>> f = tf.matmul(c, d) # matrix multiplication
>>> print(e.eval(), f.eval())
[[18. 18.]
 [18. 18.]]
[[36. 36.]
 [36. 36.]]

Tensor data types

1
2
3
4
5
6

>>> a = tf.ones(2, dtype=tf.int32)
>>> a.eval()
array([1, 1], dtype=int32)
>>> b = tf.cast(a, dtype=tf.float32)
>>> b.eval()
array([1., 1.], dtype=float32)

Tensor shapes!

1
2
3
4
5

a = tf.range(0, 8)             # shape=(8,)
b = tf.expand_dims(a, 0)       # shape=(1,8)
c = tf.expand_dims(a, 1)       # shape=(8,1)
d = tf.reshape(a, shape(4, 2)) # shape=(4,2)
print(a.eval(), b.eval(), c.eval(), d.eval())

[0 1 2 3 4 5 6 7]
[[0 1 2 3 4 5 6 7]]
[[0]
 [1]
 [2]
 [3]
 [4]
 [5]
 [6]
 [7]]
[[0 1]
 [2 3]
 [4 5]
 [6 7]]

Graphs

Any computation in TF is represented as an instance of a tf.Graph object.

Sessions

A tf.Session() object stores the context under which a computation is performed.

  • expressions are evaluated at a specific session
  • offen use an explicit context instead of a hidden one

Use an explicit context

1
2
3
4
5
6

>>> sess = tf.Session()
>>> a = tf.ones(shape=(2, 2))
>>> b = tf.matmul(a, a)
>>> b.eval(session=sess)
array([[2., 2.],
       [2., 2.]], dtype=float32)

Variables

  • Constant tensors are immutable
  • We need some mutable tensors

Here comes tf.Variable().

Creating a variable is easy enough.

1
2
3

>>> a = tf.Variable(tf.ones(shape=(2, 2)))
>>> a
<tf.Variable 'Variable:0' shape=(2,) dtype=float64_ref>

What about to evaluate the variable a?

1
2

>>> a.eval()
FailedPreconditionError: Attempting to use uninitialized value Variable_1

As we said before, TF code just describes computation, a variable should be initialized before using it.

How to use a variable? Just initialize it!

1
2
3
4
5

>>> sess = tf.Session()
>>> sess.run(tf.global_variables_initializer())
>>> a.eval(session=sess)
array([[1., 1.],
       [1., 1.]], dtype=float32)

A variable is mutable, and statful, we can assign a new value to it!

Assigning values to variables

1
2
3
4
5
6

>>> sess.run(a.assign(tf.zeros(shape=(2, 2))))
array([[0., 0.],
       [0., 0.]], dtype=float32)
>>> a.eval(session=sess) # `a` is changed
array([[0., 0.],
       [0., 0.]], dtype=float32)

The shape should match!

Placeholders

Since we already have variables, why do we need placeholders?

A placeholder is a way to input information into a TF computation graph.

*"Think of placeholders as the input nodes through which information enters TF."*

  • a placeholder is used to feed outside data into the graph
  • a variable is initialized inside the graph
  • use feed dictionary to feed data into placeholders
  • feed dictionary: tf.Tensor $\mapsto$ np.ndarray
1
2
3
4
5

>>> a = tf.placeholder(tf.float32, shape=(1,))
>>> b = tf.placeholder(tf.float32, shape=(1,))
>>> c = a + b
>>> c.eval(session=sess, feed_dict={a: [1.], b: [2.]})
array([3.], dtype=float32)

Review

  1. tensor: basic obejct in TF
  2. graph: computation language
  3. session: computation context
  4. variable: statful and assignable
  5. placeholder: “mouth” of graph :)

An Example

Linear Regression

Use synthetic toy data for the regression task. The data is generated by $$ y = wx + b + \epsilon, $$ where $\epsilon \sim N(0,1)$.

Generate the data

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(256)
tf.set_random_seed(256)

# Generating sythetic data
N = 100
w_true = 5
b_true = 2
X = np.random.rand(N, 1)                          # shape=(100, 1)
noise = np.random.normal(scale=0.1, size=(N, 1))
Y = w_true * X + b_true + noise                   # shape=(100, 1)

Plot the data

Generate TF graph

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Generate tensorflow graph
with tf.name_scope("placeholders"):
    x = tf.placeholder(tf.float32, shape=(N, 1))
    y = tf.placeholder(tf.float32, shape=(N, 1))

with tf.name_scope("weights"):
    W = tf.Variable(tf.constant(4.9)) # shape=() rank-0
    b = tf.Variable(tf.constant(2.1)) # shape=() rank-0

with tf.name_scope("prediction"):
    y_pred = W * x + b # shape=(100, 1)

with tf.name_scope("loss"):
    l = tf.reduce_sum((tf.squeeze(y - y_pred))**2)

with tf.name_scope("optim"):
    train_op = tf.train.AdamOptimizer(.001).minimize(l)

with tf.name_scope("summaries"):
    tf.summary.scalar("loss", l)
    merged = tf.summary.merge_all()

train_writer = tf.summary.FileWriter('/tmp/lr-train', tf.get_default_graph())

Perform training

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# perform training
n_steps = 1000
with tf.Session() as sess:
    # initialization
    sess.run(tf.global_variables_initializer())
    for i in range(n_steps):
        feed_dict = {x: X, y: Y}
        _, summary, loss = sess.run([train_op, merged, l], feed_dict=feed_dict)
        print("step %d, loss: %f" % (i, loss))
        train_writer.add_summary(summary, i)

Thank you!