Useful Stuff

This is a collection of miscellaneous shell scripts, commands, config stuff, and generally just things that you wouldn't want to have to search for online again after going through all the trouble of finding them in the first place.

Videos and image sequences

Create movies from image sequences (needs to be tested on Mac/Windows):

• MPEG using ffmpeg (preferred for optimal cross-platform compatibility, -b is bitrate): ffmpeg -i frame%03d.png -b 2000k output.mpg
• JPEG to AVI: mencoder -ovc lavc -mf fps=25:type=jpg 'mf://*.jpg' -o outputfile.avi
• PNG to MPEG2 (framewise seekable as keyinterval set to 1): mencoder -ovc lavc -oac copy -mf fps=25 'mf://*.png' -of avi -lavcopts vcodec=mpeg2video:keyint=1:mbd=1:vqmin=2:vqmax=10:autoaspect -o outputfile.avi

Extract frames from video file:

• JPEG: mplayer -nosound -vo jpeg inputfile
• PNG (z=compression level): mplayer -nosound -vo png:z=[0-9] inputfile

see Philippe Dreuw's page for more

Mplayer

By default mplayer will resize video using bicubic kernel, which can be too blurry if you're trying to look closely at pixel values. To view a video in fullscreen mode using nearest neighbour resampling:

• mplayer -vo x11 -sws 4 -zoom input.avi

To view multiple frames of a video at the same time (tiled):

• mplayer -vf scale=160:90,tile=8:8

To convert MTS files from the Sony HDR-SR7 into something more editable (lossless compression): the following will convert to greyscale (I think the lavdopts flag needs to go before the lavcopts one to get this to work properly), uses the ffv1 codec in it's highest compression/slowest mode (CABAC,large context), uses a good, fast, general purpose deinterlacing algorithm (yadif), resizes the video and ensures that every single frame is actually encoded (noskip, softskip, harddup) and makes sure that no weird framerate conversions get applied (fps/ofps).

• mencoder -oac copy -lavdopts gray -ovc lavc -lavcopts vcodec=ffv1:coder=1:context=1:gray -vf softskip,harddup,yadif=1:0,scale=480:270 -noskip -fps 30000/1001 -ofps 30000/1001 -o output.avi input.mts

Note that to search for black frames (i.e. when synchronising to strobe flashes) you can use:

• mplayer -vf blackframe=X:Y

To prevent the annoying flicker when looping a video, make sure that the "-loop 0" flag goes at the very end of the command line.

To get the OSD to work, search for the installed fonts using

• locate .ttf

and choose one to insert into your ~/.mplayer/config file like so

• font=/etc/splashy/themes/openSUSE/VeraSans.ttf

Disk usage

Figure out what's eating up your quota

• du -h | grep "^[0-9]*M[ \t]*"

Text processing

To extract the n'th (or every n'th) line

• awk '{ if(NR==n) print $0 }'
• awk '{ if(NR%n==1) print $0 }'

Where $0 is the whole line, $1 is first column, $2 is second column etc.

Search and replace across multiple files

perl -pi -w -e 's/search/replace/g;' *.cpp

where 'search' and 'replace' are regular expressions

Use

Calling "use -L" will list all the available packages, but since it outputs to stderr you can't grep the list for whatever you're looking for. Fortunately you can redirect stderr to stdout by calling it like so (alias for convenience):

use -L 2>&1

tmk

To add a new module to your tmk configuration, edit the file $HOME/.tmk/site-config/. Here is an example (typically you would only need to specify the INCPATH and LIBS variables, and OPTIONS is useful for specifying various library-specific flags.) After this you can just specify the module name in a TMakefile to use it. Run 'tmk -reconfig' afterwards. Run 'tmk -sysinfo' to see the paths to current config files.

namespace eval cimg {
   config set DEPEND {x11 pthread fftw}
   config set INCPATH {/ubc/cs/research/imager/project/psm/Packages/Generic/CImg-1-18/include}
   config set LIBS {png z jpeg tiff}
   config set cxx::OPTIONS {cimg_strict cimg_use_xshm cimg_use_xrandr cimg_use_fftw3}
}

The variables $psm_pckg_generic and $psm_pckg/[::arch_name ""] may be used to refer to the directories /ubc/cs/research/imager/project/psm/Packages/Generic and /ubc/cs/research/imager/project/psm/Packages/Arch/SuSE10.1 respectively.

Matlab

When using mcc to compile standalone applications, you may have trouble running them if a different version of Java is being used. It seems as if Matlab uses the JRE in $JAVA_HOME when it compiles code, but then produces a script that sets paths so that it looks in {matlabroot} for a Java runtime. As of 21 April 2009, these are different versions. The workaround is to edit the run_progname.sh script that mcc outputs, and change the MCRJRE variable to point to $JAVA_HOME/lib/{amd64|i386}. For 32bit this would be/usr/lib/jvm/jre/lib/i386 and for 64bit it would be /usr/lib64/jvm/jre/lib/amd64

Producing publication-quality figures

Old information:

Export a figure to eps file. The 'save' option on the file menu of the figure window tends to behave erratically, especially if exporting very large size. To export figures for publication, use these scripts instead. Or use the laprint script for better LaTeX documents (using psfrag).

New information:

So far I've only found one to produce decent-quality matlab figures for use with pdflatex. To get axes and figure labels to use latex fonts, you need to use psfrag, which is only available through latex/dvi2ps/ps2pdf. But if you want to use pdflatex then the psfrag package is unavailable. A solution is to compile a separate pdf file using the latex...ps2pdf and then insert that into the pdflatex project. The following makefile code could be used to do this transformation. It takes a sample.m file and produces a sample.pdf image (with as-tight-as-possible cropping) that can be included in your tex file.

%.pdf: %.m
   # default size here is 12cm wide but that can be overridden by a comment
   # in the m file of the form: "% FigureWidth 20"
   # this ridiculously convoluted sequence of latex programs appears to be the
   # only way to get the bounding boxes correct. pdfcrop and epstool sometimes make
   # serious errors when given the output of latex|dvips|ps2pdf, so we have
   # to use ps2epsi.
   echo "set(0,'defaulttextinterpreter','none');" >$*_0.m
   cat $< >>$*_0.m
   echo -n "laprint(gcf,'$*','asonscreen','on','width'," >>$*_0.m
   echo -n $(shell grep FigureWidth $< | sed 's/.*[^0-9]\([0-9][0-9]*\).*/\1/') >>$*_0.m
   echo ",'keepfontprops','on');" >>$*_0.m
   matlab -nodesktop -nosplash -r "$*_0; exit;"
   echo "\documentclass{minimal}" >$*.0.tex
   echo "\usepackage{graphicx,color,psfrag}" >>$*.0.tex
   echo "\pagestyle{empty}" >>$*.0.tex
   echo "\begin{document}" >>$*.0.tex
   echo "\input{$*.tex}" >>$*.0.tex
   echo "\end{document}" >>$*.0.tex
   latex $*.0.tex
   dvips $*.0.dvi
   ps2epsi $*.0.ps
   ps2pdf -dEPSCrop $*.0.epsi
   mv $*.0.epsi.pdf $@
   rm $*_0.m $*.{0.aux,0.log,0.pfg,0.tex,0.dvi,0.epsi,0.ps,tex,eps}

SSH without typing in passwords

Run these commands to avoid having to enter your password when SSHing into any machine that can mount your home directory.

ssh-keygen -t rsa
cp $HOME/.ssh/id_rsa.pub $HOME/.ssh/authorized_keys2
chmod 0600 $HOME/.ssh/*

There will be 3 prompts, and you can just hit enter for each of them. Unless you want a passphrase (which is probably a good idea from a security point of view) in which case you'll need to run an agent on login (see http://www.cs.umd.edu/~arun/misc/ssh.html). You'll still need to type "yes" the first time you log in to a new machine, to add it to the list of known hosts. Note that this will overwrite any previous keys you may have configured. To log in from computers outside the network, you can append their public keys (id_rsa.pub) to authorized_keys2.

MDA

MDA file"> - View an MDA file

If the mda file contains an ordinary image, you can view it using

mda-toimage <file.mda PPM:- | display

- Average a sequence of (single-channel) files

mda-stack *.mda | mda-project --pre 0,0 --post '%0/%1,1' '%0+#0,%1+1' | mda-swizzle -cl 0 > average.mda

- Misc

Expression calculations are done with doubles, on a range of 0 to 1. So to increment the brighness of a standard ubyte 0..255 image by one, add 1/255.

Latex

Figures

Latex symbols can be inserted into xfig/dia diagrams. If using dia, draw the graphics and convert to xfig format first by exporting to eps and then running

pstoedit -f xfig myfile.eps myfile.fig

Start xfig in in latex font mode with

xfig -specialtext -latexfonts -startlatexFont default

Add the labels as ordinary text in latex format (eg. $\delta$). Adjust the font size in xfig until it looks about right. Note that the final symbols will obviously take up a different amount of space as the xfig text, but they will start at the same (upper, I think) left coordinate (assuming it's set to left-alignment). Now, export to "combined PDF/Latex (both parts)" which will produce two files (it's best if you give your target filename a .pdf suffix). You might have to install the linux transfig package to do this. The file.pdf will contain the graphics, while file.pdf_t will contain the latex code necessary to render the text (and to include the image part automatically). You might need to hand edit the .pdf_t file to fix the file path. Add the image to your latex document the usual way, but with \input instead of \includegraphics.

\begin{figure}[tbh]
   \centering
   \resizebox{\textwidth}{!}{\input{myfile.pdf_t}}
   \caption{My caption}
   \label{fig:myfile}
\end{figure}

Making nice labels for figures

You can generate a rasterised version of arbitrary LaTeX code with this script: https://www.cs.ubc.ca/wiki/core/pub/Imager/UsefulScripts/tex2png

Converting xfig, inkscape and gnuplot figures

Here is makefile code to convert .fig, .gpl and .svg files to .pdf for inclusion in latex documents with pdflatex Note that when converting xfig files, the ONLY formats that correctly support latex strings are the various combined ps/pdf/latex formats and metapost (mp). The former will require that you use psfrag/latex/ps2pdf or that you compile a separate pdf file like in the gnuplot case. It's just simpler to use metapost.

%.pdf: %.fig
   fig2dev -L mp $< >$*.mp
   mptopdf --rawmp --latex $*.mp
   mv $*-0.pdf $@
   rm $*.{log,mp,mpx,0}

%.pdf: %.svg
   inkscape -f $< --export-pdf=$@

%.pdf: %.gpl
   echo "set terminal epslatex colour" >$*.0.gpl
   echo "set output \"$*.tex\"" >>$*.0.gpl
   cat $< >>$*.0.gpl
   gnuplot $*.0.gpl
   epstopdf $*.eps
   echo "\documentclass{minimal}" >$*.0.tex
   echo "\usepackage{amsmath,graphicx,color}" >>$*.0.tex
   echo "\pagestyle{empty}" >>$*.0.tex
   echo "\begin{document}" >>$*.0.tex
   echo "\input{$*.tex}" >>$*.0.tex
   echo "\end{document}" >>$*.0.tex
   pdflatex $*.0.tex
   pdfcrop $*.0.pdf
   mv $*.0-crop.pdf $@
   rm $*.{0.gpl,0.aux,0.log,0.pdf,0.tex,eps,tex}

EPS files

If the bounding boxes on EPS figures you export are too large, you can manually correct them. Load the eps file in gv and position the mouse curson at the bottom left (lx,ly) corner, and then the top right (rx,ry) corner. The coordinates will be displayed in the box at the top left of the window. Open the file in a text editor and find the %%BoundingBox line. Change the 4 values after it to read lx ly rx ry. The first two are xy coords of lower left corner, next pair is coords of upper right. 0,0 is bottom left of page. For more info, see online.redwoods.cc.ca.us/instruct/darnold/staffdev/Tips/tip10.pdf

Alternatively convert to pdf and use the pdfcrop perl script (it's floating around on the web somewhere). Sometimes it makes mistakes and crops off too much. I suspect the problem lies with the source eps file in those cases, and can be fixed by using ps2epsi instead of ps2eps. The epsi files contain binary rather than human-readable data and are larger, but pdf files generated from them are the same as those produced from eps files.

Profiling Applications

Valgrind

Valgrind is a useful set of tools for profiling linux applications. The main tools are:

• Memcheck - for memory management problems
• Cachegrind - a cache profiler
• Callgrind - an extension to cachegrind with extra information about call graphs
• Massif - a heap profiler
• Helgrind - a thread debugger

One benefit of these tools is that they are easy to use on any executable, even one that was compiled with optimization. No debug info is needed. The latest version is installed in the PSM package system. Type:

use valgrind

More information is available at http://www.valgrind.org

Examples

1. Finding bottlenecks

If your program is myApp with arguments arg1 arg2, then run the following command:

valgrind --tool=callgrind myApp arg1 arg2

Note, it will run up to 50 times slower than normal. Upon completion, valgrind will output a file called callgrind.out.pid where pid is the process ID. Use KCachegrind to view the contents of this file.

KCachegrind

KCachegrind is a visualization tool for the output from valgrind. The latest version is already installed on the lab machines. Just run it as follows:

kcachegrind callgrind.out.pid

The visualization is straightforward. For more documentation, see the sourceforge page here.

Matlab

Did you know: Matlab has a built-in profiler. Under the Desktop menu, or using the "profile on" command. Very useful.

Project Webpages

Instructions for putting up project webpages are posted here: http://www.cs.ubc.ca/labs/imager/tr/submitting.html Note, however, that the paths described on these pages are according to the old format. For a more permanent link, you should use the following path prefix: /ubc/cs/research/imager/imager/local/generic/web/{tr,video}

Compute Servers

Got a big job to run? Look no further than the ICICS compute servers! As an added bonus: avoid annoying your labmates by slowing down their machines! Remember to use full /ubc/cs/... path names in your scripts, and write all data to the home, imager or RAID filesystems, since local disk access is not available. Note that unfortunately a few of these cannot access the RAID space. Whether this is a permanent thing, or related to the intermittent access problems we've been having, is unknown.

To see the list of machines

netgroup icicscompute

Cascade is intended for interactive sessions. Running long processes is best done on granville or columbia if you need a Solaris host, or okanagan, begbie, or kokanee if you want a Linux host.

Sun Grid Engine - quick user guide

https://bugs.cs.ubc.ca/cgi-bin/twiki/view/BETA/SunGridEngine#Administration

A Quick Tutorial for beta_grid Cluster with Sun N1 Grid Engine 6.0

http://www.cs.ubc.ca/~xkliang/Beta_Grid_Cluster.html

hdrgen

Shruthi's hdrgen reference page is: /imager/people/shruthi/Research/hdrgen/hdrgen.html

Photocopier Code

The photocopier code is 8541. There is no password. This is the code for Imager, and it should work on all dept copiers. 1831 is for copying for courses.

How to Convert PowerPoint Slides to Images

* Change the slide resolution

• http://www.techsmith.com/learn/camtasia/6/record/powerpoint-slides-as-images.asp