Skip to content

Concepts

TensorBoard is a visualization toolkit for machine learning experimentation. TensorBoard allows tracking and visualizing metrics such as loss and accuracy, visualizing the model graph, viewing histograms, displaying images and much more. TensorBoard’s main features include:

  • Visualizing the graph of a TensorFlow model
  • Tracking model metrics like loss and accuracy
  • Examining histograms of weights, biases, and other components in the ML workflow
  • Displaying non-tabular data, including images, text, and audio
  • Projecting high dimensional embeddings into a lower-dimensional space

Important

Tensorboard can be used with both Tensorflow and PyTorch ML frameworks.

Scalars

The machine learning process necessitates the tracking of different metrics related to a model’s performance. This is important to spot any problems quickly and to determine if a model is overfitting, among other things.

Using the Scalars Dashboard of TensorBoard, we can visualize these metrics and debug the model quickly. The procedure for logging involves three simple steps:

  • Create a callback
  • Specify a directory to log the data
  • Pass the callback when calling the fit method

The Tensborboard scalar shows changes in loss and accuracy after every epoch (i.e. when an entire dataset is passed through a neural network both forward and backward propagation). It can also be used to track other scalar values such as learning rate and training speed.

Tensorboard Scalar

The approach described above works for most scenarios, but what if we want to log a custom scalar that is not readily available? For scenarios like this, we can use the Summary API of TensorFlow. This API is used for writing summary data which can later be used for visualization and analysis. Let’s look at an example where we want to use a simple sine wave as the scalar we want to display on TensorBoard.

# Specify a directory for logging data
logdir = "./logs"
# Create a file writer to write data to our logdir
file_writer = tf.summary.create_file_writer(logdir)
# Loop from 0 to 199 and get the sine value of each number
for i in range(200):
    with file_writer.as_default():
        tf.summary.scalar('sine wave', np.math.sin(i), step=i)

Using the scalar method from tf.summary, we can log pretty much any scalar data we want. We are not limited to losses and metrics when using TensorBoard. Shown below is output for the above example.

Custom Scalar

Images

When working with image data, users may want to view the data to look for any problems or just review samples to ensure data quality. With TensorBoard’s Image Summary API, this can be performed easily. Let's review an example to see how we can achieve this.

Let’s first load the data using the datasets module of TensorFlow.

# Load and normalize MNIST data
mnist_data = tf.keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist_data.load_data()
X_train, X_test = X_train / 255.0, X_test / 255.0

After this, we will take the first image and visualize it. We will reshape it first before passing it to the file writer.

# Reshape the first image
img = np.reshape(X_train[0], (-1, 28, 28, 1))
# Specify a directory for logging data
logdir = "./logs"
# Create a file writer to write data to our logdir
file_writer = tf.summary.create_file_writer(logdir)
# With the file writer, log the image data
with file_writer.as_default():
    tf.summary.image("Training data", img, step=0)

After logging the image data, let us go back to the dashboard. If we look at the Images tab, we will see the image we selected is displayed on the dashboard.

Tensorboard Images

Graphs

All TensorFlow models can be seen as a computation graph. It can sometimes be difficult to see the architecture of a model by looking at the code alone. A visual representation can benefit us greatly by making it easy to see how a model is structured. This will also ensure that the architecture we are using is what we intended or designed.

Shown below is the model definition 

# Define the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
])

Now, let us create a TensorBoard callback and use it when we train the model.

# Create a callback
tf_callback = tf.keras.callbacks.TensorBoard(log_dir="./logs")
# Pass in the callback when fitting the model
model.fit(X_train, y_train, epochs=5, callbacks=[tf_callback])

Once training is complete, click on the Graphs tab. The first thing we see is the op-level graph of the model. This graph shows the model architecture layer by layer. This is important to see if our model is correct and each of the layers is what we intend them to be. Note that the graph is inverted, with data flowing from the bottom to the top.

Tensorboard Graphs

Distributions and Histograms

TensorBoard Distributions and Histograms are another great way to track the progress of a model. Once training is completed, you will notice that several tabs show up on TensorBoard Let's look at an example for Distributions.

Tensorboard Distribution

The above set of graphs shows the tensors that make up the model. On the horizontal axis of each graph, we see the epoch number, and on the vertical axis, we see each tensor’s value. The graphs show how these tensors change over time as training progresses.

The darker areas show where the values spent the most amount of time. If we are concerned that our model weights are not updating properly at each epoch, we can spot those problems using this tab.

Now, click on Histograms. Notice that these graphs show a different view of the tensors in the model. We see that each graph has five histograms stacked on top of each other, representing each of the five epochs we have trained. They show similar information regarding where the tensor weights tend to center around. Once again, this is useful for debugging misbehaving models.

Tensorboard Histograms

Text

Text is a commonly used type of data when creating machine learning models. It can be hard to visualize text data. With TensorBoard, users can visualize text data easily with the Text Summary API. As a simple example, we’ll work with the text "Hello World".

# Our sample text
sample_text = "Hello world!"
# Specify a directory for logging data
logdir = "./logs/text/"
# Create a file writer to write data to our logdir
file_writer = tf.summary.create_file_writer(logdir)
# With the file writer, log the text data
with file_writer.as_default():
    tf.summary.text("sample_text", sample_text, step=0)

Once training is complete, let us look at TensorBoard's Text tab. We see the text we entered in the Text tab.

Tensorboard Text