Caffe Logging and Loss Plotting

Over the past few months I have been working with the neural network toolbox, Caffe. It is pretty full featured, but niceties such as logging and plotting are pretty much missing.

Usually I train with Caffe from Python, since it allows me to initiate layers with just a bit of Python code.

python solve.py <solverstate file> 2>&1 | tee -a solve.log

For some reason, Caffe prints everything out to stderr instead of stdout, this is why I redirect it using 2>&1. Tee splits stdout to both a file (which is appended thanks to -a) and also displays it to the screen. If I didn't use tee I would just end up with another terminal for tail -f.

The output looks something like:

I1222 22:21:16.484805 25190 solver.cpp:236] Iteration 444940, loss = 0.0230301
I1222 22:21:16.484844 25190 solver.cpp:252]     Train net output #0: loss = 0.035717 (* 1 = 0.035717 loss)
I1222 22:21:16.484851 25190 sgd_solver.cpp:106] Iteration 444940, lr = 1e-05
I1222 22:21:24.420318 25190 solver.cpp:236] Iteration 444960, loss = 0.0216933
I1222 22:21:24.420359 25190 solver.cpp:252]     Train net output #0: loss = 0.0269941 (* 1 = 0.0269941 loss)
I1222 22:21:24.420367 25190 sgd_solver.cpp:106] Iteration 444960, lr = 1e-05

We want to plot the iteration loss. This can be done with a short shell script, with the help of GNU Plot. I have also experimented with doing this with MATLAB but it executes much slower.

#!/bin/bash

data=`\
cat solve.log | \
grep ", loss = " | \
awk '{ print $6,$9 }'`

echo "$data" | gnuplot -p -e "set term xterm; plot '-' with lines title 'loss' "

The "set term xterm" is optional. If you don't know what it means you should probably remove it. I execute this script inside xterm because I am often working over a very slow internet connection. This allows plotting directly in xterm tektronix mode, which is much faster for me than X Forwarding. The only disadvantage is that I can't move around the plot or zoom in.

Anyway, that's how I plot my loss. Hope this is helpful to someone.

Tags: caffe linux