Caffe Logging and Loss Plotting
Over the past few months I have been working with the neural network toolbox, Caffe. It is pretty full featured, but niceties such as logging and plotting are pretty much missing.
Usually I train with Caffe from Python, since it allows me to initiate layers with just a bit of Python code.
python solve.py <solverstate file> 2>&1 | tee -a solve.log
For some reason, Caffe prints everything out to stderr instead of stdout, this is why I redirect it using 2>&1. Tee splits stdout to both a file (which is appended thanks to -a) and also displays it to the screen. If I didn't use tee I would just end up with another terminal for tail -f.
The output looks something like:
I1222 22:21:16.484805 25190 solver.cpp:236] Iteration 444940, loss = 0.0230301 I1222 22:21:16.484844 25190 solver.cpp:252] Train net output #0: loss = 0.035717 (* 1 = 0.035717 loss) I1222 22:21:16.484851 25190 sgd_solver.cpp:106] Iteration 444940, lr = 1e-05 I1222 22:21:24.420318 25190 solver.cpp:236] Iteration 444960, loss = 0.0216933 I1222 22:21:24.420359 25190 solver.cpp:252] Train net output #0: loss = 0.0269941 (* 1 = 0.0269941 loss) I1222 22:21:24.420367 25190 sgd_solver.cpp:106] Iteration 444960, lr = 1e-05
We want to plot the iteration loss. This can be done with a short shell script, with the help of GNU Plot. I have also experimented with doing this with MATLAB but it executes much slower.
The "set term xterm" is optional. If you don't know what it means you should probably remove it. I execute this script inside xterm because I am often working over a very slow internet connection. This allows plotting directly in xterm tektronix mode, which is much faster for me than X Forwarding. The only disadvantage is that I can't move around the plot or zoom in.
Anyway, that's how I plot my loss. Hope this is helpful to someone.