VLF Receiver Toolkit - Notes


  Index


  Installation


Ubuntu

Installation is very easy on Ubuntu/Debian systems. Just run the following apt-get commands (prefix with sudo if you haven't enabled the root login):

   apt-get install libasound2-dev
   apt-get install libvorbis-dev
   apt-get install libflac-dev
   apt-get install libx11-dev
   apt-get install libpng12-dev
   apt-get install libxpm-dev
   apt-get install libncurses5-dev
   apt-get install libforms2
   apt-get install libforms-dev
   apt-get install libshout3-dev
   apt-get install libsamplerate0-dev
   apt-get install libfftw3-dev
   apt-get install sox
   apt-get install gnuplot

That takes care of all the packages that the toolkit depends on. Then install the toolkit with the instructions below.

If you're not running Ubuntu or some other flavour of Linux that uses apt-get, then see the section Prerequisite Packages below.


Raspberry Pi

Installation is straightforward on the RPi. If running the Raspbian operating system, the following apt-get commands will install all the prerequisites (prefix with sudo if you're not using a root login):

   apt-get install libasound2-dev
   apt-get install libncurses5-dev
   apt-get install libshout3-dev
   apt-get install libflac-dev
   apt-get install libsamplerate0-dev
   apt-get install libfftw3-dev
   apt-get install libpng12-dev
   apt-get install libx11-dev
   apt-get install libforms2
   apt-get install libforms-dev
   apt-get install sox
   apt-get install gnuplot

Then continue with the installation instructions below.

Note that, USB isochronous data transfer (as used by soundcards) does not work properly on the Raspberry Pi (as of Jan 2013). USB data packets are silently dropped and this renders the capture side of the device unusable (typically 1% of the audio packets are lost).

A work-around is to add dwc_otg.speed=1 to /boot/cmdline.txt which makes all USB (and the ethernet) run at USB 1.1 speed (12Mbit/sec). This enables USB sound capture to work reliably with many cards, but usually prevents the USB keyboard and/or mouse from working.


Installation

Download the tgz file and unpack. cd down into the package directory and run

   ./configure
Configure will report if any prerequisite packages are missing. Some of these can be ignored given suitable options to configure.

On the Raspberry Pi, if you are using Raspbian or Archlinux you should enable the use of hardware floating point, configure with

   ./configure --with-hardfloat

The command ./configure --help will show all the configuration options. When configuration is successful, continue with the installation,

   make
   make install
Programs will be installed in /usr/local/bin.

If you get a make error on OpenBSD, you may need to do

   LDFLAGS="-lpthread -lspeex -lz -lossaudio"
   export LDFLAGS
and then repeat the configure and make install.

If make reports an undefined OV_ECTL_COUPLING_GET then you must upgrade your vorbis library libvorbis to version 1.3.2 or later, http://www.xiph.org/vorbis/.


Prerequisite Packages

You can run ./configure first to see which packages, if any, are missing. Maybe you don't need them and can turn off their requirement with configuration options mentioned below. All the prerequisites are easy to obtain and install.

alsa/oss
The operating system will probably already have ALSA installed and configured. Older systems, or non-linux, or those few cards that don't have working ALSA support but do have 4Front OSS support, may require installation of OSS.

Some systems such as Ubuntu don't install development files. If you find that the package configures itself for OSS but you know you have ALSA drivers installed, try

   apt-get install libasound2-dev

The package can be compiled without soundcard support if required. This may be appropriate if the installation is purely for post-processing of data and is not required to read from a soundcard. In this case, configure the package with --without-soundcard and program vtcard will not be compiled.

fftw
Download from http://www.fftw.org/ and install.

libvorbis
Required only for program vtvorbis. Disable this requirement with --without-vorbis. Download from http://www.xiph.org/vorbis/ and install. Ubuntu users can just do

   apt-get install libvorbis-dev
Ubuntu may also need
   apt-get install libshout3-dev

libFLAC
Required only for program vtflac. Disable this requirement with --without-flac. To install flac, download from http://flac.sourceforge.net/download.html, or on Ubuntu, do

   apt-get install libflac-dev

xforms
This is used to make control panels for some of the programs. Download from http://xforms-toolkit.org/ and install. Ubuntu users may have to also do

   apt-get install libjpeg62
   apt-get install libjpeg62-dev

libsamplerate
Required for sample rate conversion vtresample. In the unlikely event that you don't need this, configure with --without-resample. Otherwise, download from http://www.mega-nerd.com/SRC/ and install.

ncurses
This is used by vtstat to report the state of a stream. You will probably already have the libraries installed, but some Linux installations don't include the ncurses.h header file. On Ubuntu you may have to do

   apt-get install libncurses5-dev
Otherwise, disable the requirement (and program vtstat) with configure option --without-curses.

libshout
This package is used by vtvorbis -u to uplink data to an icecast server. You can bypass this requirement with --without-shout, or download from http://www.icecast.org/download.php and install.

X11
Used along with xforms for interactive displays on some of the programs. These are optional. Remove with --without-x11; On Ubuntu you may have to do

   apt-get install libx11-dev
   apt-get install libpng12-dev
   apt-get install libxpm-dev

Other Packages Some other packages that are very useful to have, are Sox and gnuplot. On Ubuntu, install these with

   apt-get install sox
   apt-get install gnuplot


  Streams and Buffers


Signal streams

The programs in this package exchange data with one another using a simple packeted data format. Each packet is timestamped and contains sample rate calibration information. The data is streamable, meaning that a program can begin work in the middle of a stream and does not need to see the start. Signal streams can be exchanged through a choice of files, fifos, lock-free buffers, and network connections.

Stream timestamps are preserved through the toolkit programs and data storage and retrieval. Timestamp resolution is 1nS and the accuracy can be better than 1uS if a good PPS is available from a GPS.

Signal samples within the stream have a choice of formats: 16 bit and 32 bit signed integers, or double precision floating point.

A stream can contain any number of signal channels and the samples are interleaved into frames.


Lock-free buffers

These are circular buffers established using shared memory and appear as virtual files under the directory /dev/shm. Each buffer has just a single input process but any number of processes can consume data from the buffer. The input process will never block waiting for consumers. Consumer processeses will block and sleep until a packet of data is available. If a consumer process cannot keep up with the flow of data, the buffer will eventually overrun and consumers will be able to detect this and take appropriate action.

Lock free buffers are used where a stream needs to be split n-ways, or where upstream processing needs to be protected from downstream hold-ups.


Stream Naming

Stream files and named pipes use normal unix pathnames. Lock-free buffers use a buffer name which is prefixed with an '@' character to distinguish it from a pathname. Network connections begin with a '+' character.

Examples:-
   /tmp/foo          A file or named pipe;
   @bar              A lock-free buffer called 'bar';
    -                Standard input or standard output FIFOs;
   +foo,41720        Send output by TCP/IP to port 41720 on host 'foo';
   ++foo,41720       As above but with persistent network connection;
   +41720            Listen for input on TCP/IP port 41720;
   ++41720           As above, but listen again if connection dropped;

All streams names can be qualified with additional syntax. Input streams can specify which channels within them are to be used. The name is followed by a colon and a comma-separated list of channel numbers (counting from 1).

Examples:-
   /tmp/foo        Use all channels from input /tmp/foo;
   @bar:4          Use only the 4th channel;
   /tmp/foo:1,3    Use first and third channels;
   +51300:2        Listen on port 51300 and use the 2nd channel;

Output stream names may be qualified to specify a data format and, in the case of lock-free buffers, a buffer size.

   /tmp/foo        Output to file or pipe /tmp/foo using 8 byte floating point;
   /tmp/foo,i2     Use 2 byte signed integers instead;
   @bar,i4         Create a buffer of 4 byte signed integers, default 10 seconds length;
   @bar,20,i4      Create a 20 second buffer of 4 byte signed integers;
   @bar,20         Create a 20 second buffer using default f8 format;
   +bar,51300,i2   Send to port 51300 at hostname 'bar' using 2 byte signed integers;

The available data format options are:

f8  8 byte floating point (ie double precision);
f4  4 byte floating point (ie float);
i4  4 byte signed integer;
i2  2 byte signed integer;
i1  1 byte signed integer;

All channels within a stream will use the same data format. The default format is f8. When using integer data formats, the programs scale sample values so that +/-1.0 corresponds to the maximum value represented by the integer size. You must ensure that output signals from programs directed into integer formatted streams don't exceed +/-1.0.

Network connections can be made to be persistent by beginning the stream name with ++ instead of +, for example ++bar,51300. In this case if a connection drops, or a connection fails to be established, the sending end will keep retrying and the receiving end will continue to listen. This prevents a processing pipeline from collapsing if a network connection has a problem, and is useful in start-up scripts when the order in which hosts are booting is not guaranteed.


  General Notes


Commands

Many of the toolkit programs have a lot of command line options, too many to remember. All the programs will accept an option -? which outputs a usage message and option summary. Options and arguments follow the usual unix command line syntax of space separated fields and getopt conventions. Where some options need sub-arguments these are comma separated.


Messages

All of the programs will accept one or more -v options to indicate the level of verbosity of messages. With no -v, only significant events are reported. Using -v will give some additional info, -vv for even more detail, and -vvv is only useful for software debugging.

Messages will go to a logfile if one is specified (-L), and to the stderr stream unless the program is put into background with a -B option.


Log files

Many of the programs will produce a log file if given a -L logfile option. The logfile argument gives the pathname to the required log file. The log file will receive all the messages selected by the -v options. Log files are written with an open/append/close sequence for each message, therefore they can easily be removed or renamed for log rotation.


Network connections

Network connections specified by the + or ++ syntax are compatible with the netcat utility. For example

   bar $   vtstat +41720
   foo $   vtcat @source +bar,41720
has the same effect as
   bar $   vtstat +41720
   foo $   vtcat @source | nc bar 41720
or
   bar $   nc -l 41720 | vtstat
   foo $   vtcat @source | nc bar 41720
but requires one less process in the pipeline at each end.

When using the ++ syntax to make a persistent connection, the sending end will wait 5 seconds if the connection drops and then try to reconnect, retrying every 5 seconds. A receiving end will wait 1 second after loss of connection and go back to listening. Data will usually be lost during a network interruption and the receiving end will detect a timing break on the stream. When a connection is reestablished, the stream parameters (format,sample rate) must be the same as before, otherwise the receiving end will exit with an error.

With network connections set up using the + syntax, the sending end always makes the connection and the receiving end always listens. If you want the receiving end to make the connection, eg to contact and downlink data from a server, you must use netcat.


Package version

The installed version can be queried with

   vtstat -V


Test scripts

The installation can be tested using the scripts in the test subdirectory. These exercise some of the functions of the software. The test scripts will create some temporary files /tmp/vttest*.

After doing a make install, run all the test scripts with

   cd test
   ./runall


Time specifiers

Many of the programs accept a timestamp or timestamp range introduced by a -T option. Timestamps are given in ISO format,

   YYYY-MM-DD_HH:MM:SS.ssss...
or as a real number of seconds since 1970-01-01 00:00:00 and are always in UT regardless of local time zone. The fractional part of the second has nanosecond resolution. Timestamps may be truncated, but at least YYYY-MM must be given. The special values 'today' and 'yesterday' can be used for the previous two midnights and the value 'now' is replaced by the current time when the command line is parsed. The following are valid timestamps,
   2010-12-24_23:59:58.479312
   2010-12-24_23:59:58
   2010-12-24_23:59
   2010-12-24_23
   2010-12-24
   1293235198.479312

A range of time is referred to in these notes as a 'timespec' and has the forms

   start,end
   start,
   ,end
where start and end are timestamps. If start is missing, the start timestamp defaults to the beginning of data. If end is missing, the default is the end of data or the current time.

end can be supplied as a real number beginning with a '+'. This indicates a number of seconds offset from the start time. For example the following two timespecs both select the same period,

   2011-03-27_12:15,2011-03-27_13:15.25
   2011-03-27_12:15,+3600.25

The absence of a timespec usually implies the selection of all available data.


Complex coefficients

Some of the programs accept complex numbers on their command line or configuration, such as the -c option of vtmix or the coefficients in the eqmap of vtfilter. This package uses a uniform syntax for these complex coefficients.

Coefficients may be given by their real and imaginary components, for example
   1.5+0.2j       1.5-0.2j      -1.5+2e-1j
Note the 'j' follows the imaginary part. The coefficient can have no imaginary part, or no real part, for example
   2.5            2.5j          -4.1e-2j    j      -j
Alternatively, coefficients may be given by their magnitude and phase lead in degrees, as follows
   3.5/45        -1.5/350       4.5e-3/-10
In either form, coefficients must not have any embedded spaces.

Absolute phase

Several of the programs report either 'absolute phase' or complex amplitudes. For example vtsid records absolute phase, vtwspec and vtnspec report complex amplitudes. The absolute phase and the phase angles of the complex amplitudes are referenced to a cosine wave having a phase of zero at zero timestamp, ie at 1970-01-01_00:00:00.0

In other words, if vtsid reports that a carrier at frequency F has absolute phase P degrees, then that carrier can be reconstructed with a sinusoidal factor of

   cos(2*pi*F*T + P*pi/180)
where T is a timestamp.

Periodic waveforms produced by the signal generator vtgen are also referenced to a phase of zero at zero timestamp.

A consequence is that all these programs can be stopped and restarted and they will continue to report or generate the same phase.


  Signal Processing Modules


vtcat

Copy input stream to output stream.

   vtcat [options] [input [output]]

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
 -T timespec Extract specified time range
 -S secs     Skip the first secs seconds of input
 -E secs     Transfer only specified number of seconds, then exit
 -p          Pad timing breaks with zeros
 -a secs     Adjust timestamp by adding an offset of secs seconds
examples:
   # Select only channels 1 and 3 from the input stream
   vtcat input:1,3 output

   # Swap left and right on a stereo stream
   vtcat input:2,1 output

   # Change a stream data format to 4 byte integers
   vtcat input output,i4

   # Get the stream from buffer @raw on host foo and load it into a buffer
   # of the same name on localhost
   ssh -nc blowfish foo vtcat @raw | vtcat - @raw
  
   # Send data from channel 2 of buffer @raw to port 9876 on host foo,
   # using 2 byte integer format, and maintain the connection
   vtcat @raw:2 ++foo,9876,i2

   # Listen for data on network port 9876 and transfer anything received
   # into a buffer @remote.  Continue listening if the connection drops.
   vtcat ++9876 @remote

   # Extract a particular time range of data from a stream
   vtcat -T 2010-08-01_14:11:40,2010-08-01_14:33:20 input output

   # Extract 30 seconds of data starting at the given time, and make
   # a histogram from it
   vtcat -T 2010-08-01_14:11:40, -E30 input | vtstat -h bins=100

   # Subtract 9uS from the timestamp of the stream
   vtcat -a -9e-6 input output

This program is used for changing the format of a stream, or selecting particular channels from a stream, or selectively transferring data between pipes, named fifos, and lock-free buffers.

If no output stream is specified, stdout is used. If no input or output stream is given, vtcat processes data between stdin and stdout.

The timestamp of the stream may be adjusted with a -a option. The argument specifies the number of seconds to add to the stream's timestamp.

One thing that vtcat doesn't do is concatenate streams. The regular cat utility is adequate for that.

vtcat in conjuction with xinetd can be used to implement a simple stream server. For details see Stream server


vtmix

Additive mixer

   vtmix [options] -c matrow [-c matrow] ...  [input [output]]

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile

 -c matrow   Define an output channel by its row of matrix coefficients
examples:
   # Turn single channel stream into 2 channel stream by duplication
   vtmix -c1 -c1 input output

   # Produce RH circular polarisation response from orthogonal loops
   # ch1 = E/W, ch2 = N/S, loop polarities in phase given north-east signal,
   # delay N/S by 90 deg and add to E/W
   vtmix -c1,-j input output

   # Or instead, advance E/W by 90 deg and add to N/S
   vtmix -cj,1 input output

   # Synthesise single loop oriented 040 deg from the above orthogonal loops
   # sin(40) = 0.643, cos(40) = 0.766
   vtmix -c0.643,0.766 input output

   # Produce RH and LH circular response as a two channel output stream
   vtmix -cj,1 -c1,j input output

   # As above, but include the original orthogonal channels to produce a
   # four channel output stream
   vtmix -cj,1 -c1,j -c1,0 -c0,1 input output

vtmix mixes the channels in the input stream according to the coefficients given by the -c matrow options. One output channel is produced for each -c. The matrow argument is a comma-separated list of real or complex coefficients and must have one coefficient (which may be zero) for each of the channels in the input stream. For example,

    vtmix -c1.2+0.5j,-0.4-0.1j ...
produces an output channel formed from
   ch1 * (1.2 + j0.5) + ch2 * (-0.4 - j0.1)
If any of the coefficients contain imaginary parts, the mixing is performed in the frequency domain by means of an overlap-add Fourier transform. If only real coefficients are given, the mixing is carried out in the time domain.

Coefficients may be given as complex numbers or by magnitude and phase. See Complex coefficients for the general syntax of the coefficients. Up to 32 input and output channels can be mixed.


vtmult

Multiplicative mixer

   vtmult -f freq [options] [input [output]]

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile

 -f freq     Local oscillator frequency, Hz
examples:
   # Shift SAQ at 17.2kHz down to 400Hz 
   vtmult -f16800 @source | vtmix -c1,-j - @dest

Each channel of the input is multiplied by a quadrature local oscillator to produce I and Q output signals. The oscillator phase is determined from the input stream's timestamp and is zero degrees at 1970-01-01 00:00:00. The mixer is balanced, so there is no output at the oscillator frequency unless there is a DC component in the input signal.

There are two output channels, I and Q consecutively, for each input channel.

Sum or difference frequencies can be selected by following vtmult with vtfilter. Upper sideband can be selected by following vtmult with vtmix -c 1,-j and lower sideband is obtained with vtmix -c 1,j.


vtjoin

Combine two or more streams, aligning by timestamp and correcting for sample rate differences.

   vtjoin [options] input1 [input2] ... output

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
examples:
   # Join together the stream coming through named pipe /tmp/str1
   # with the stream on stdin, selecting just the first two channels
   # from stdin.  Output to /tmp/joined
   vtjoin -- -:1,2 /tmp/str /tmp/joined

   # Turn a single channel stream into a 2 channel stream by duplication,
   # sending output to stdout
   vtjoin @test @test - 

The input streams are merged together into a single output stream having a number of channels equal to the sum of the selected input channels. All the input streams must have the same nominal sample rate. vtjoin uses the timestamp information to align the inputs, and corrects for sample rate differences between the input streams by duplicating or discarded samples as necessary.

The output stream always has the nominal sample rate, ie has sample rate calibration factor of 1.0. vtjoin does not 'look' at the signals passing through it, and relies entirely on the incoming timestamps to do the alignment.

vtjoin must have an output stream specified, it does not default to stdout. Up to twenty input streams can be joined.


vtsid

Monitor for sudden ionospheric disturbances.

   vtsid [options] -c config_file input_stream

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile

This program monitors VLF signals within the stream and logs various signal parameters that may be used to detect ionospheric disturbances. Either the whole band can be logged as an array of frequency bins, or specific channels can be logged, or both. Data is saved into a database maintained by vtsid and the program vtsidex is used to extract records.

Parameters which may be recorded against each bin or channel are:

If the input stream contains channels derived from orthogonal loops, the bearing (modulo 180 degrees) and relative phase angle between loops can be logged. If an E-field channel is also available, the bearing can be logged modulo 360 degrees by one of three methods. Elevation, polarisation and ellipticity can also be logged.

If the input stream is timestamped with sufficient accuracy, eg using vttime and GPS PPS, the absolute signal phase can be logged, mod 360 for CW signals, or mod 180 for MSK signals.

The program is controlled by a configuration file specified by the -c option. An example configuration file vtsid.conf is included in the package. Use this as a template, copy and edit to your requirements.


vtfilter

General purpose filter.

   vtfilter [options] [input [output]]

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
 -e eqmap    Load equalisation map
 -g gain     Gain factor  (default 1.0)
 -n bins     Number of frequency bins (default 16384)

 -a notchspec   Set automatic notch parameters
                notchspec consists of the following
                comma separated parameters:
                th=threshold,  (default 6.0)
                bw=width, (notch width, default 1 bin)

 -h filterspec  Apply a filter
 -h lp,f=corner,p=poles    Butterworth low pass
 -h hp,f=corner,p=poles    Butterworth high pass
 -h bp,f=center,w=width    Brick wall bandpass
 -h bs,f=center,w=width    Brick wall bandstop

examples:
   # Run an automatic notch filter combined with 5 pole high-pass
   # with 500Hz corner frequency, and 3 pole low-pass with 8kHz
   # corner, and an overall gain boost of factor 4.
   # Input from stdin and output to stdout.
   vtfilter -a th=7 -h hp,f=500,poles=5 -h lp,f=8000,poles=3 -g4

The input signal is transformed to the frequency domain and multiplied by an array of complex filter coefficients, and is then transformed back to the time domain using an overlap-add scheme. The filter coefficients are the product of all filter responses composed from the -h filterspec options.

An automatic notch filter, intended for removal of mains hum and harmonics, is activated with a -a notchspec option. This filter monitors the mean amplitude in each frequency bin and if a bin mean exceeds the average of its neighbours by a factor exceeding the threshold, the offending bin and some of its neighbours will be set to zero.

A -n bins option can be used to specify the number of frequency bins to use in the Fourier transform. The resulting resolution is given by sample_rate/(2 * number_of_bins). For best results with hum removal using the automatic notch, the resolution should be set to around 0.5Hz or so. The number of bins can be any number but powers of two are the most efficient.

An equalisation map can be applied with -e eqmap, where eqmap is a text file containing a list of frequencies and complex coefficients. This mechanism is intended for use in equalising the amplitude and phase response of separate receivers so that their signals can be coherently combined. The map file consists of one row per frequency. Each row must start with the frequency and be followed by a list of complex coefficients, one for each of the channels in the input stream. For example, a two channel input stream would require records in the format:

   freq coeff_chan1 coeff_chan2
Frequencies are given in Hertz and the file can contain any number of entries, which must be in ascending order of frequency. vtfilter interpolates logarithmically between eqmap records to obtain complex coefficients for each frequency bin, such that the completed transfer function is represented on a Bode plot by straight lines joining the eqmap points.

The equalisation coefficients default to unity at DC and the Nyquist frequency, so it usually necessary to include at least these two points in the map file.

To inspect the computed filter coefficients, run vtfilter with the option -d1. The program will output a table of coefficients in eqmap format, one row for each frequency bin, and then exit. The coefficients are the product of all the filters and eqmap specified.


vtevent

Monitor for natural radio events - whistlers and risers.

   vtevent [options] -d outdir input_stream

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile
 -p polarspec  Input channel assignments
 -X            Use X display
examples:
   # Read soundcard into buffer @eraw
   vtcard -Bvv -g1 -d hw:0,0 -r 48000 @eraw,20,i2

   # Filter and event detect
   vtfilter -a th=6.0 -h lp,f=10000,poles=2 -h hp,f=400,poles=2 @eraw |
     vtevent -v -L /tmp/vtevent.log -d /raw/events

This program uses a combination of principal component analysis and Hough transform to detect VLF signals having reasonably narrow bandwidth along with rising or falling frequency. Falling frequency signals are identified as whistlers if they conform to the expected shape of a whistler curve as defined by the low frequency Eckersley approximation. A dispersion range of 12.2 to 114 is examined by the Hough transform.

If a -d outdir option is given, the detected events are saved under outdir. For each event, a thumbnail spectrogram, raw data, and an information text file, are saved in files named by the event's timestamp.

Event detection thresholds are automatically set, relative to the background noise floor of the VLF input signal. The input signal should be pre-filtered to remove hum, as in the example above. If the sample rate is higher than 48k samples/sec, it is more efficient to resample to 48k/sec or lower before applying vtevent. A sample rate of 32k/sec gives the best results.

The -X produces a display of the analysis processes. This is intended only for testing and tuning, not for normal operation.

The option -p polarspec is required if the input consists of more than one channel. It indicates which channels are E-field and which are H-field and their orientations. It allows vtevent to estimate the apparent bearing, polarisation, and elevation of the whistler's incident signal. polarspec is a comma-separated list with one entry per input channel. Entries are numeric to indicate a loop orientation, or the letter 'E' to indicate vertical E-field. For example, with

   vtevent -v -d /datadir -p8,96,E
the program expects a 3-channel input with the first channel being a loop oriented on 8/188 degrees, the second channel a loop on 96/276 degrees, and the third channel is the vertical E-field.


vttoga

Measure TOGA and other parameters of sferics.

   vttoga [options] input_stream

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile
 -p polarspec  Input channel assignments

 -F from,to    Frequency range to analyse
               (default 4000,17000)

 -h threshold  Trigger threshold (default 0.1)
 -r rate       Adjust trigger threshold to maintain
               this rate of triggers per second

 -e            Output spectrum and waveform data
 -d outdir     Output directory (default stdout)
 -G seconds    Output file granularity (default 3600)

 -c            Calibration mode, trigger on UT seconds

 -N count      Capture this many pulses, then exit
 -E seconds    Operate on this many seconds of input, then exit

 -T timestamp  Analyse a single pulse beginning at this
               timestamp
examples:

   # Set an initial threshold of 0.1 and then adjust it automatically to
   # average 10 triggers per second.   Send the output to files under
   # directory /data/togas and start a new file on 10 minute boundaries.
   vttoga -h 0.1 -r10 -d /data/togas -G600 @source

   # Capture 100 calibration pulses from a GPS injected into the VLF
   # receiver input.  Record the spectrum and waveform of each.
   vttoga -ec -N100 @source > datafile

   # Log the TOGAs and bearings from a 3-axis receiver. ch1 has azimuth
   # 102 deg and ch2 has azimuth 357 deg.  ch3 is an E-field signal
   vttoga -r10 -p102,357,E @source > datafile

Sferics and other short pulses are analysed to measure their time of group arrival (TOGA). Triggering is by rate of change of amplitude and occurs when the difference between consecutive input samples (on any channel) exceeds the threshold specified by a -h option. When triggered, the program analyses 1.2mS of signal beginning 0.2mS before the trigger point. By default the band 4-17kHz is used. The lower bound avoids the cut-off frequency of the dominant propagation mode. The upper bound is restricted to avoid strong MSK signals.

If -r rate is given, the program adjusts the threshold to maintain the specified average rate of triggers per second. With multiple input channels, all channels are analysed given a trigger on any channel and the computed TOGA is the amplitude-weighted average of the TOGAs of all the channels.

vttoga operates continuously on the input stream unless -T is given, in which case a single sferic is analysed using the timestamp as a trigger time. The timestamp should point to a couple of hundred uS before the start of the pulse. vttoga requires triggering to occur before the TOGA, therefore if the timestamp is too late, no pulse will be analysed. If the timestamp is too early, accuracy will be reduced.

Using -E and/or -N, the program terminates after processing the given seconds of input stream or the specified number of triggers.

If a -p polarspec is given, the bearing of the sferic will also be measured. polarspec indicates which channels are E-field and which are H-field and their orientations. polarspec is a comma-separated list with one entry per input channel. Entries are numeric to indicate a loop orientation, or the letter 'E' to indicate vertical E-field. The loops do not need to be orthogonal. The bearing is computed for each frequency bin and the output bearing is the average weighted by the bin amplitudes.

Mains hum and MSK signals and other strong continuous signals should be removed before the signal reaches vttoga, such that the sferics dominate the signal when examined with vtscope.

Output consists of 'H' records with the following format:

   H 1329179203.193116 5.447e-02 6672 7.2e-01 240.4
            |             |       |      |      |
          TOGA           RMS    range residual bearing

The bearing field will only be present if a -p polarspec option has been given. The RMS field is the square root of the average signal power summed over all the input channels. Range in km is estimated from the dispersion of the sferic using a simple Earth-Ionosphere propagation velocity model, and will only have a chance of being reasonable if the input signals have had their phase equalised (see vtfilter -e eqmap). If the program cannot get a range estimate, a zero will be output.

With a -e option, additional records are output following the 'H' record. 'S' records describe the pulse spectrum with one record per bin:

   S 10105.263 3.993e-01 -46.464 1.629e-01 -47.558 ...
        |          |        |        |        |
    frequency  amplitude  phase  amplitude  phase  ...
                 ch1       ch1     ch2       ch2
The phase is the unwrapped phase relative to zero phase at the TOGA at the center frequency of the analysis band (-F freqspec) option. 'T' records describe the time domain waveform using one record per signal sample:
   T -2.978825e-04 3.9439e-02 -1.7892e-02 ...
         |            |            |
      time offset  amplitude    amplitude ...
                     ch1          ch2
The time offset is seconds relative to the TOGA and the amplitudes are the input sample amplitudes, one for each channel. Following the 'S' and 'T' records is a single 'E' record:
   E
This marks the end of the records for that sferic and closes off the data set started by the 'H' record. The output format is intended to be easy to parse using programs such as awk to separate and plot the data sets.

By default the stream of 'H' and optional extended records is sent to the standard output. If -d outdir is given, the output is sent to files created under outdir,

   outdir/YYMMDD-HHMMSS
and a new file is started whenever the timestamp crosses a boundary set by the -G option.


vtrsgram

Produce reassigned spectrogram.

   vtrsgram [options] [input]

options:
 -v            Increase verbosity
 -oa           ASCII data output
 -opng         png image output (default)
 -ox           X11 interactive display

 -b bins       Number of frequency bins
 -s stepping   Time axis stepping factor (default 2)

 -a factor     Prune low amplitude points
 -r factor     Prune long range points
 -p factor     Mixed partial derivative pruning

 -g gain       Image (brightness) factor (default 1.0)
 -m scale      Image magnification factor (default 2.0)

 -n            Produce ordinary non-reassigned spectrogram
 -h count      Use homomorphic vertical EQ

 -w window     Specify window function:
                  cosine (default), blackman, hamming,
                  nuttall, hann;
examples:
   # Render the whistler sample using 320 bins, advancing the FT frame by
   # half the FT width.  Render as a png file four times the size of the
   # STFT grid and discard the weakest 30% of points.
   vtrsgram -b320 -s2 -g4 -m4 -a0.3 /raw/whistlers/1321774508 > 1321774508.png

   # As above but ASCII output,  -g and -m don't apply.
   vtrsgram -b320 -s2 -a0.3 /raw/whistlers/1321774508 > 1321774508.dat

This program produces a short time Fourier transform of its input stream on a regular F,T grid of bins frequency values and time intervals of 2 * bins/(stepping * sample_rate) seconds. The amplitudes of the grid cells are then reassigned to new non-grided F,T points according to the reassignment method of Auger and Flandrin. The T value is modified according to the local group delay of the STFT cell and the F value is reasigned to the instantaneous frequency of the cell.

If -oa is given, the reassigned points are output as 3-column ASCII numeric values: F, T and RMS amplitude.

Without -oa, the reassigned points are mapped to a new grid of size scale times the original STFT grid and the result is output as a png file. The -g gain option allows the brightness to be adjusted, with larger values of gain causing high amplitude points to saturate and low amplitude points to become more visible.

The -s stepping option indicates how far the Fourier transform window advances for each T step of the initial STFT spectrogram. With stepping of 1, the frame advances by the FT width and each input sample is used only once. A stepping of N advances the frame by 1/Nth of an FT frame and each input sample is used N times.

Weak amplitude points can be pruned with -a prune where prune is a number between 0.0 (no pruning) and 1.0 (everything pruned). Reassigned points are ranked in amplitude order and the weakest prune fraction are discarded. A typical value for prune would be 0.5 which starts to have a significant effect on the image.

Points lacking local support can be pruned with -p factor which examines each point's mixed partial derivative of phase with respect to time and frequency, as per Nelson's method. Smaller values of factor will remove more points. -p0.6 gives quite significant pruning, -p2.0 gives slight pruning.

Range pruning is activated with -r factor and this removes points where the reassignment moves the point by more than factor times the STFT cell size. This exploits the redundancy present in the STFT and a point pruned by this option will usually be reinstated more accurately by reassignment of a closer STFT cell. A typical value for factor would be 2.0 and larger factors have less effect.

Intensity variation in the vertical direction can be levelled off using the homomorphic filtering option -h count where count is a small integer. The initial STFT is transformed to a 2D log spatial frequency domain and the lowest count number of vertical spatial frequencies are removed. The effect is to remove multiplicative gain variation with frequency. Typical values of count range from 1 to 20 and 10 is often sufficient.


vtblank

Impulsive noise blanker

   vtblank [options] input output

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
 -c          Clip impulses at the threshold
             (default is to blank the output)
 -d secs     Dwell time, seconds
             (default is 0.005 seconds)
 -h thresh   Threshold amplitude
 -a factor   Automatic threshold factor
 -t secs     Time constant for moving average
             (default 100 seconds)
 -w chanspec Operate only on these channels
examples:
   # Apply a threshold of 1.2 times the mean, with zero dwell and 1 second
   # time constant.   This is a reasonable setting when looking for narrow-
   # band signals underneath VLF background.
   vtblank -a1.2 -d0 -t1

This program removes impulsive noise such as sferics and switching transients which exceed a given threshold, and is intended for use when looking for weak continuous signals.

Impulses may be blanked (output set to zero) or clipped (output limited at the threshold). When the input signal exceeds the threshold, the output is blanked or clipped for a retriggerable duration given by the -d dwell time option. Without -d, only those samples exceeding the threshold are modified.

A -h option specifies the threshold amplitude. Usually it is better to use the -a option which sets the threshold automatically to be the given factor higher than the exponential moving average noise floor of the signal. The time constant of this moving average can be set by a -t option.

By default, if the threshold is exceeded on any channel, all the channels are blank or clipped. This is usually desirable for further signal processing. This is disabled if -i is given, in which case each channel is examined and blanked or clipped independently of the others.

The -w chanspec option indicates which of the selected input channels the program should work on. By default, all selected input channels are blanked. chanspec is a comma-separated list of channel numbers.


  Storage and Retrieval


vtwrite

Write stream to disk files

   vtwrite [options] input datadir

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
 -G seconds  Output file granularity (seconds)
             (default 86400 = 1 day)
examples:

The input stream is recorded into a set of disk files which are stored in the directory datadir. The format of the files is identical to that of the stream, which means the data files can later be used directly as input to VT modules.

Output files are named by their start time. For example, if datadir is /raw, an output file will have a name such as /raw/110624-195425.

A new output file is started whenever the stream timestamp crosses a boundary specified by the -G file granularity option. By default (86400 seconds), this starts a new file at midnight. A setting of -G 3600 will start a new file on each hour.

A new file is also started if a timing break is detected on the input stream, or if the vtwrite program is stopped and restarted.

vtwrite stores data in the file in the same format as that of the input stream. Therefore, if you want to store samples in say i2 format, you must provide an input stream in i2 format, eg

   vtcat input -- -,i2 | vtwrite /raw/vt.

As mentioned above, the data files can be used directly as input to VT programs, or the vtread program can be used to extract data, which allows specific time ranges to be read, seamlessly across multiple files.


vtread

Read a stream from disk files created by vtwrite

   vtread [options] datadir

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile
 -T timespec   Restrict records to a range of times.  Default is all records

vtread is used to selectively extract stream data from a set of files produced by vtwrite. datadir specifies the directory containing the file set. The program will scan the files in datadir to extract as much data as possible within the time range given by -T timespec.

See Time specifiers for the description of timespec.


vtgen

Signal generator.

   vtgen -r rate [options] output

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile
 -r rate       Specify sample rate (no default)
 -c chans      Number of channels (default 1)
 -g gain       Output gain (default 1.0)
 -T start,end  Range of timestamps  
 -t            Throttle output rate to approximately real time
 -s a=amplitude,f=freq,p=phase
               Generate a cosine wave, amplitude specifies peak
 -n a=amplitude
               Generate normally distributed white noise, amplitude
               specifies mean
 -u a=amplitude
               Generate uniformly distributed white noise, amplitude
               specifies the peak
 -p a=amplitude,f=freq,p=phase,d=duty
               Generate rectangular pulses, amplitude specifies level
 -m a=amplitude,f=freq,p=phase,b=bitrate
               Generate an MSK signal
 -d v=level    Generate DC level
 -a seed       Seed the random number generator
 -i infile     Specify input file for time source
examples:
   # Send an 18kHz sine wave out through the ALSA soundcard port hw:0,0 at
   # 48k samples/sec stereo
   vtgen -p a=0.5,f=18000 -r 48000 -c2 | vtraw -ow | aplay -

   # Simulate the MSK signal from NSY at 45.9kHz, using vtmix to produce
   # orthogonal loop signals with the MSK bearing 155 degrees.
   vtgen -r192000 -m a=0.1,f=45900,b=200,p=38.2 -t |
     vtmix -c -0.960 -c 0.423 | vtsid -c sid.conf

vtgen creates a stream containing a mix of signals, pulses, and noise. Multiple signal component options -s,-n,-p can be given and the signals are mixed together into the output stream. The timestamp of the stream is taken from the system clock unless a -T or -i option is given. Output is throttled to approximately real time rate if -t is given, otherwise data is generated as fast as possible.

Amplitude options should be given in the range 0 to 1, frequency is specified in Hertz, duty cycle must be in the range 0 to 1, phase is given in degrees.

The phase is an absolute phase taking account of the timestamp. Zero phase is at 1970-01-01 00:00:00.

MSK can be generated with a -m option, which is useful for testing vtsid. Set b= twice the value of the sid monitor's br= setting.


vtfm

Frequency modulator/demodulator.

   vtfm [options] input output

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile

 -m            Modulator
 -d            Demodulator (default)

 -F hertz      Specify carrier frequency (modulator only)
 -k index      Modulation index, Hz per unit
examples:
   # Demodulate an FM sensor signal, carrier at 12kHz, 1kHz deviation produces
   # one unit output amplitude
   vtmult -f12000 @source |
      vtfilter -h lp,f=2000,poles=8 |
      vtfm -d -k1000 - @signal

This program extracts a signal which has been frequency modulated onto a carrier, for example with a voltage-to-frequency convertor. It is intended for capture of low frequency sensor signals.

The input stream must carry pairs of baseband I/Q signals, one pair per sensor channel, and the input must be band-limited to the width of the FM signal. This typically involves the use of vtmult followed by vtfilter.

The modulation index given by the -k option indicates the frequency deviation in Hertz required to produce an output amplitude of 1 unit.

With -m and -F options, the program functions as a modulator. This is only useful for generating test signals.

With suitable allocation of the soundcard spectrum, several low frequency sensors can be mixed together, each with its own carrier frequency, into a soundcard input channel.


vttime

Refine the timestamp of a stream.

   vttime [options] input output

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile
 -c chan     Specify input channel containing timing signal
 -m method,params  Specify timing method and parameters
examples:
  # Re-time the signal in buffer @raw using the positive-going baseband PPS
  # signal on channel 2.  The centroid of the PPS pulse occurs 250uS later than
  # the UT second.
  vttime -c2 -m ppsbase+,c=250e-6 @raw @output

A new timestamp is assigned to the stream, and the stream is resampled to the exact sample rate. Output samples are synchronous with UT, that is, there is a sample having a timestamp of the exact second and all the other samples are at intervals of 1/sample_rate.

A timing signal is presented to the program on one of the input channels. The -m options specifies the timing method to use and indicates the expected form of the timing signal, as follows:

Limited use is made of the timestamp of the input stream. It is used only to get a rough estimate of when to expect the next timing pulse and should be accurate to better than 0.5 seconds otherwise the second will be ambiguous.


vtresample

Change the sample rate of a stream.

   vtresample -r rate [options] [input [output]]

options:
  -v          Increase verbosity
  -B          Run in background
  -L name     Specify logfile
  -r rate     Convert to this sample rate
  -q qual     Conversion quality:
               qual = 0  fastest (default)
                    = 1  medium
                    = 2  best
  -g gain     Gain factor (default 1.0)
examples:

The input stream is converted to the new sample rate specified by the mandatory -r rate option. rate must be an integer, but can have any ratio to the input sample rate.

The program will not run with a sample rate ratio more than about 250 in either direction and will terminate with 'src_process error'. In that situation, perform the conversion in two stages.


  Input/Output Programs


vtcard

Initialise and read data from a soundcard.

   vtcard [options] [outstream]

options:
 -d device   Specify the OSS or ALSA device to use
             (default /dev/dsp or hw:0,0)
 -c chans    Number of channels to read (default 2)
 -b bits     Bits per sample (default 16)
 -r rate     Sample rate (no default, mandatory option)
 -g gain     Output gain (default 1.0)
 -u          Disable sample rate tracking
 -v          Increase verbosity
 -B          Detach from terminal and process group to become
             a daemon program
 -T stamp    Apply a dummy timestamp when reading stdin
 -A opts     Specify buffering options,
               -A b=buffer_size,p=period_size
             sizes in bytes, either or both b= or p= can be given
             If not specified, reasonable defaults are used
 -L name     Specify logfile
examples:
   # Simple test using the default soundcard in stereo
   vtcard -vv -r48000 @test
   vtstat @test

   # Read /dev/dsp2 at 192k/sec 4 byte samples, 2 channels, into
   # a lock-free buffer called dsp2raw.  The buffer is made 30 seconds
   # long and uses 4 byte integer values.
   vtcard -r 192000 -d /dev/dsp2 -b32 @dsp2raw,30,i4

   # Read ALSA soundcard, specifying a large buffer size suitable for 192k/s
   # Run in background, output to buffer called @hew
   vtcard -Bvv -A b=262144,p=2048 -d hw:1,0 -b32 -r 192000 @hew,20,i4

This program initialises the soundcard for input and reads data into the specified output stream. It is strongly recommended to use a lock-free buffer as the output stream in order to prevent soundcard overruns.

If vtcard is run with superuser permissions, it will assign itself a real-time priority and will lock itself and its output buffer into memory. Under these conditions vtcard will only block while waiting for soundcard data and soundcard overruns are extremely unlikely.

It may take a minute or two before vtcard starts to write data to its output stream. During this time it makes an initial measurement of the soundcard's sample rate. After this, vtcard continues to monitor the sample rate (against the system clock) and includes this calibration information in its output stream.

The data is timestamped using the system clock, after correcting for the latency reported by the driver. If a soundcard overrun does occur, vtcard initiates a reset of the card and there will be a break in the output stream while the sample rate and timestamp are re-evaluated.

If the card cannot run at the requested sample rate, vtcard will use the closest available rate.

If the option -d- is given, vtcard will read raw PCM samples from its standard input. In this mode, an initial timestamp is taken from the system clock, and thereafter the sample rate is assumed to be exact. A -T option can be given to supply an alternative start time.

The program will choose a suitable period or fragment size and buffer length. These can be overriden with a -A option.

For further information, see Soundcards


vtvorbis

Encode to, or decode from, ogg/vorbis and multiplex the timestamp data as a second logical stream within the ogg container.

   vtvorbis [options] [-e|-d] name

options:
 -v          Increase verbosity
 -e          Encode
 -d          Decode
 -B          Run in background
 -L name     Specify logfile
 -p          Expect/generate only a pure ogg/vorbis stream
             (default is to multiplex with a timestamp stream)

 encoding options:
 -q factor   Specify VBR encoding with this quality factor
             factor range -0.1 to 1.0
 -b kbps     Specify CBR encoding at this kbps per channel
             (default is CBR, 64kbps per channel)
 -i          Independent encoding of each channel

 decoding options:
 -E secs     Decode only specified number of seconds, then exit.

 network options:
 -u method,server,port,mount,passwd
                Uplink to icecast server
 -n server,port Send over network, without protocol, to port on server
 -n port        Listen on port, without protocol
 -k             Retry failed uplink connections
                (default is to exit if connection drops)
 -t             Throttle uplink to sample rate
             (default is to encode and uplink as fast as possible)
examples:
  # Send data from @test to a buffer with the same name on host xyz
  vtvorbis -eq0.5 @test | ssh -c blowfish xyz vtvorbis -d @test

  # Filter and stream channel 1 of the raw VLF signal in buffer @eraw,
  # as pure ogg/vorbis to Icecast server
  vtfilter -a th=7 -h hp,f=500,poles=2 -h lp,f=10000,poles=2 -g4 @eraw:1 |
  vtresample -r 32000 |
  vtvorbis -ep -q0.4 -ktu shout,67.207.143.181,80,/vlf1,xxxxx

  # Downlink ogg/vorbis from a stream server.  Let wget make the http
  # connection as it knows all about proxies, authentication, etc,
  # and use aplay to listen to it.
  wget -q -O- http://67.207.143.181/vlf1 2> /dev/null | 
  vtvorbis -dp | vtraw -ow | aplay -

  # Send vorbis stream to server abelian.org port 1234
  vtvorbis -ei -q0.4 -kn abelian.org,1234 @source

  # Listen on port 1234 for incoming vorbis connection
  vtvorbis -d -kn 1234 @destination 

By default, vtvorbis -e multiplexes the timestamp and sample rate calibration as a second stream within the ogg container. This allows the receiving vtvorbis -d to reconstruct the VT stream, preserving timestamps across the link. Media players and other decoders are supposed to ignore any unrecognised streams in the container, but some don't and therefore when sending to a decoder other than vtvorbis, a -p option may be a necessary encoder option.

The -E option is useful in scripts which sample a stream and need to capture a specific length of data for analysis. The output will be approximately the requested number of seconds in length - output is stopped at the end of the vorbis page following the specified end point.

A -i option forces independent encoding of the channels. The default is to allow the vorbis encoding to exploit the redundancy between channels, which is usually desirable for efficient encoding of stereo but introduces spurious correlation of background noise and weak signals between channels. Therefore, when encoding independent streams, a -i option is necessary to avoid this problem.

-n server,port can be used with -e to send encoded data to a remote server. The server should use -d -n port to listen for the incoming connection. The connection has no connection or wrapping protocol and is equivalent to piping the vorbis stream through a netcat connection.

When uplinking over a WAN, a -k option is recommended. This will ensure that a dropped connection is retried until eventually reestablished. vtvorbis -ek will retry a failed connection every 20 seconds. While the link is down, encoded data will be discarded so that the upstream processing pipeline is not choked. Without -k, the encoder will exit if the uplink fails. -k can be used with both -u for protocol connections, and with -n for simple connections.

Recommended mode of operation is VBR with a quality factor of 0.3 or 0.4 for a single channel of 32k samples/sec.

LAN network connections can use the netcat (nc) program to good effect when the hosts involved are safely behind a firewall. For WAN connections, ssh is usually necessary for connections between vtvorbis. In this case, use blowfish encryption as this is the most efficient.

Caveat vtvorbis doesn't work at sample rates below 15k/sec or above 50k/sec.


vtflac

Lossless compression/decompression using FLAC, preserving timestamps.

   vtflac [-e|-d] [stream]

options:
 -v          Increase verbosity
 -e          Encode
 -d          Decode
 -B          Run in background
 -L name     Specify logfile
examples:
   # Archive an i2 format stream file
   vtflac -e /raw/111225-000000 > /arc/111225-000000.fx

   # Unarchive the same file
   vtflac -d /raw/111225-000000,i2 < /arc/111225-000000.fx

vtflac -e applies loss-less compression to the stream data using FLAC and multiplexes the compressed audio with the timestamp information. The result is a file which is readable only by vtflac -d.

Audio data is converted to 16 bit (i2) resolution by the encoder, and if the original file is i2 or lower resolution, the decoded samples are recovered exactly.

If no input or output buffer or filename is given, stdin and stdout are assumed and the program operates in a pipeline.

The files produced by vtflac -e will not be playable by a FLAC player because of the multiplexed timestamp data. This program is intended for archival of stream files but can also be used to transfer streams across a network connection if bandwidth is at a premium. The FLAC encoder is set to run at the maximum compression available and a file of VLF data is typically reduced to about 50% of the original (i2 format) file size.


vtain

Read A/D converters on Beaglebone

   vtain -r rate -c mask [options] [outstream]

options:
 -c mask     Bit mask 0..255 specifying which A/D channels to read
 -r rate     Sample rate (no default, mandatory option)
 -g gain     Output gain (default 1.0)
 -z offset   Input offset 0..4095 (default 0)

 -v          Increase verbosity.
 -B          Detach from terminal and process group to become
             a daemon program
 -L name     Specify logfile
examples:
   # Read AIN1 at 200 samples/second, scaling the A/D range of 0..4095
   # to output range -1 .. +1
   vtain -c1 -r200 -z2048 -g2 @output

   # Read all 8 A/D converters at 20 samples/second
   vtain -c255 -r20 @output

This program reads samples from the A/D convertor nodes in the directory /sys/devices/platform/omap/tsc. Up to 8 channels can be read, specified by the bit mask argument to the -c option.

By the default, the 0..4095 range of the A/D is mapped to 0..+1 in the channel data.


vtdata

Read data from ASCII source

   vtdata -r rate -c chans [options] [outstream]

options:
 -c chans    Number of channels in input file (no default)
 -r rate     Sample rate (no default, mandatory option)
 -g gain     Output gain (default 1.0)

 -v          Increase verbosity.
 -B          Detach from terminal and process group to become
             a daemon program
 -L name     Specify logfile
examples:
   # Read two channels from a two-column text file
   vtdata -c2 -r32000 -g2 @output < source.txt

This program reads data from a flat ASCII text data file. Maximum record width is 1024 characters. The data is timestamped by the current time at the start of the program.


vtrtlsdr

Take data from RTL2832U dongle.

   vtrtlsdr [options] buffer_name

options:
  -v        Increase verbosity
  -B        Run in background
  -L name   Specify log file
  -d device Device number (default 0)
  -d ?      List available devices
  -r rate   Sample rate 1000000 to 3200000

  -F hertz  Tuner frequency, Hertz
  -g gain   Tuner gain (dB, default 0 = auto)
  -g ?      List available gain settings
  -q        Invert the Q signal

  -u        No sample rate tracking
  -T stamp  Start time when using -u
            (default is to use system clock)
examples:
   # Play FM broadcast station at 102.7Mhz
   vtrtlsdr -u -F 102.7e6 -r 1000000 |
      vtresample -r200000 |
      vtfm -k1e5 | vtresample -r48000 |
      vtraw -ow | aplay -

   # Record meteor pings in upper sideband at 55.25Mhz, into 1 hour files
   vtrtlsdr -F 55.249e6 -r1000000 |
      vtresample -r8000 | vtmix -c1,-j -- - -,i2 |
      vtwrite -G3600 /data/meteor

This program uses librtlsdr to retreive raw I/Q data from the tuner dongle. -F specifies the center frequency in Hz and bandwidth is determined by the selected sample rate.

Output is a 2-channel stream with channel 1 carrying the I signal and channel 2 carrying Q.

vtrtlsdr may take a minute or so to measure the sample rate of the device before it starts to deliver data to the output stream. This function is turned off with a -u option.

Allowed sample rates are 1e6 to 3.2e6 samples/second.

To include vtrtlsdr you must configure with --with-rtlsdr and install some prerequisite libraries. For more info see RTL2832U


vtsdriq

Take data from SDR-IQ receiver.

   vtsdriq [options] buffer_name

options:
  -v        Increase verbosity
  -B        Run in background
  -L name   Specify log file
  -d device Device node (default /dev/ttyUSB0)
  -r rate   Sample rate (default 196078)
            Rate must be one of
               8138, 16276, 37793, 55556
               111111, 158730, 196078

  -F hertz  Frequency, Hertz

  -g gain   RF gain in dB (default 10)
            Valid range -8 to +34
  -a        Insert -10dB attenuator
  -i        IF gain in dB (default 0)
            Allowed values 0, 6, 12, 18, 24

  -q        Invert the Q signal
  -u        No sample rate tracking
  -T stamp  Start time when using -u
            (default is to use system clock)
examples:
   # Receive upper sideband 14.0 to 14.0277 Mhz without tracking the
   # sample rate and display the spectrum
   vtsdriq -F14e6 -r55556 -g10 -i0 -q -u |
      vtmix -c1,-j | vtspec

   # Receive a single channel streamm of baseband VLF with sample rate
   # tracking
   vtsdriq -F0 -r55556 -g10 -i0 @vlf,i2

This program initialises and reads data from an SDR-IQ receiver connected via USB. -F specifies the center frequency in Hz and bandwidth is determined by the selected sample rate.

vtsdriq may take a minute or so to measure the sample rate of the device before it starts to deliver data to the output stream. This function is turned off with a -u option.

If the frequency given by -F is non-zero, the output is a two channel I/Q stream with channel 1 carrying the I signal and channel 2 carrying the Q signal. If a frequency of zero Hertz is selected, the output is just a single channel of real data.

To include vtsdriq you must configure with --with-sdriq.


vtraw

Extract audio from a stream.

   vtraw [options] [stream]

options:
 -v          Increase verbosity
 -B          Run in background
 -L name     Specify logfile

 -ob      16 bit PCM output (default)
 -oa      Produce ASCII output
 -ow      WAV output containing 16 bit PCM
examples:
   # Produce a WAV file from a whistler event file
   vtraw -ow 1325151287.vt > 1325151287.wav

   # Play a stream in real time
   vtraw -ow @signal | aplay -

   # Extract a short piece of signal for analysis
   vtread -T2010-09-15_12:31:02.47,+0.02 /raw | vtraw -oa > sferic.dat

vtraw outputs the audio content of its input stream, by default in raw 16 bit PCM with interleaved samples. With a -oa option, multi-column ASCII output is produced with the timestamp in column 1 and the audio channels in the remaining columns.

WAV output format is produced with the -ow option.


  Display and Plotting Programs

Programs for displaying signals and producing graphs.

vtstat

Report statistics
   vtstat [options] input

options:
 -E secs     Examine only specified number of seconds, then exit.
 -h hspec    Produce a histogram. hspec has the form
               max=max,bin=bins
 -a aspec    Report amplitude statistics.  aspec has the form
               r=secs
             r= specifies output record interval in seconds
 -i          Report summary statistics

Without options, vtstat uses the terminal to display some stream statistics in real time.

A histogram is generated if -h hspec is given. bins specifies the number of amplitude bins, max gives the max sample amplitude.

The -a option extracts the first three moments of the rectified input signal, averaged over a period given by the r= parameter. If r=0 is given, a single output record is produced for the entire input stream. The -a option allows amplitude probability distributions to be measured using standard metrics such as Vd or parameterisation by Gaussian, Rayleigh or Weibull distributions.

Option -i produces summary statistics to stdout.


vtscope

An oscilloscope to display a stream in the time domain.
   vtscope [options] input

options:
 -v          Increase verbosity
 -display display  Specify the X host and display number
                   (By default, the DISPLAY environment variable is used)
examples:
   # Display the signal passing through buffer @raw
   vtscope -v @raw

   # Merge the signals in buffers @raw1 and @raw2 and display them all
   vtjoin @raw1 @raw2 - | vtscope -v

   # Display just channels 1 and 3 in @raw
   vtscope -v @raw:1,3

This program uses the X display to make an oscilloscope and control panel.

Most of the controls should be obvious. Clicking the mouse on the display will report the timestamp and amplitude of the point clicked. Drag the display left or right to scroll through the capture buffer, or use the scroll bar. Button 'Arm' puts the scope into single shot mode, 'Run' sets continuous mode. If neither of these is pressed, the scope retains the captured buffer for analysis.

The display can be resized just by dragging a corner of the window.

One of the channels, or UT, can be selected as a trigger source. If UT is selected the scope triggers on the UT second mark - this is useful for examining timing pulses. A fixed pre-trigger period of 50mS is used, ie the trigger point will be at 50mS on the time axis.

The 'Plot' button puts the current waveforms into an interactive gnuplot. Press 'q' in the gnuplot window to dismiss. 'Save' stores the data into a file in ASCII format with space separated fields. The first column is the absolute timestamp, second column is the time offset into the capture buffer, and the remaining columns are the channel amplitudes.


vtspec

A spectrum analyser
   vtspec [options] [input]
examples:
   

This program uses the X display to make a spectrum analyser and control panel. If an input stream is not given, stdin is assumed.

The display can be scrolled and shifted by dragging with the mouse or by working the scroll bar and thumbwheels. Drag a corner to resize. A selection of window functions is available. Use 'Rect' for accurate peak level measurements but some of the others, eg 'Nuttall' or 'Blackman' have much lower scalloping effects but introduce amplitude errors that vtscope doesn't correct for.

The 'Plot' button puts the spectrum data into an interactive gnuplot. Press 'q' in the gnuplot window to dismiss. 'Save' stores the data to an ASCII text file with space separated fields. The first column is the frequency and the remaining columns are the bin amplitudes for each channel. Phase information is not stored.


vtcmp

Compare two channels in various ways.
   vtcmp -m mode [options] input

options:
 -S secs     Skip seconds of input before starting
 -E secs     End after accepting seconds of input

 -m type     Type of comparison.  type is one of
              cor     Cross correlation (default)
              pd180   Phase difference, mod 180
              pd360   Phase difference, mod 360 

 -F freqspec              
 -w seconds  Half-width over which comparison is made
 -N count    Average this number of input buffers
 -r          Envelope correlation
examples:
   # Outputs the correlation coefficient between channels 3 and 4
   # of the input stream, up to a range of +/- 10mS.
   vtcmp -m cor -w 0.01 source:3,4 > datafile

   # Produce an envelope correlation, smoothing the envelope by
   # ignoring components above 500Hz.
   vtcmp -m cor -w 0.01 -r -F,500 source:1,2

This program makes various types of comparison between two input channels, and writes the results to stdout. Output is multi-column plain ASCII. Exactly two input channels must be specified. -S and -E options can be used to limit the time range of the input signal to be analysed. If not given, the program continues to the end of input. The program averages the comparison in units of time given by the -w option.

With -m cor the cross correlation is averaged over the duration of the input signal. The range of time offsets to be reported is given by the -w argument.

Option -F freqspec can be used to restrict the frequency to be examined. freqspec takes the form start,end where start and end are frequencies in Hertz. Either can be omitted and start defaults to zero and end defaults to the Nyquist frequency.

A -r option causes both input channels to be rectified by squaring and the resulting 'power' signals are presented to the cross-correlation. A -F option should be used to low-pass filter the envelope as in the example above.


vtnspec

A narrow band spectrum analyser.
   vtnspec [options] input

options:
 -B              Run in background
 -L logfile      Specify logfile
 -v              Increase verbosity

 -N frames       Average up to this many transform frames, then exit
 -c              Coherent averaging of frames until end of input
 -a              Non-coherent averaging of frames until end of input

 -f hertz        Center frequency
 -w width        Frequency width in Hertz
 -r resolution   Frequency resolution - bin width in Hertz

 -p              Pad with zeros across timing breaks
 -W window       Specify window function:
                     rect (default), cosine, blackman, hamming,
                     nuttall, hann
examples:
   # Check sample rate accuracy against RSDN-20 on 11.9kHz using a 10uHz
   # bin size, using a sferic blanker to improve S/N ratio
   vtblank -a1.0 -d0 input | 
     vtnspec -cv -f 11904.7619 -w 0.02 -r 0.00001 > data.txt

This program analyses its input signal in the frequency domain using an array of Goertzel filters. It is especially suited to examining a narrow piece of spectrum or to implement overlapping bins.

The frequency range specified by the -f and -w options is spanned by Goertzel filter bins separated by the resolution given with the -r argument. -f specifies the center frequency and the program always uses an odd number of bins so that there is always a bin positioned at the center frequency. If a width of zero is specified, a single bin will be transformed, positioned at the center frequency. A transform is completed after 1/resolution seconds. By default, the program will issue an output and exit after one transform is completed.

If -a or -c is given, vtnspec will average successive transform outputs non-coherently or coherently, respectively, until end of input, or until the number of frames given by a -N option have been processed.

Output is multi-column space separated ASCII, one row for each bin, with the following format:

   ...
   1000.00000000 9.269461e-05 1.213293e-05 6.610408e-05 -1.023862e-05 ...
        |             |           |            |            |
   bin frequency   ch1_real    ch1_imag     ch1_rms      ch2_real     ...
If -a has been given to specify non-coherent averaging, the output contains just the RMS column for each channel:
   ...
   1000.00000000 2.330445e-05 1.539060e-05 ...
        |             |           |
   bin frequency   ch1_rms     ch2_rms     ...

vtnspec is typically used with up to a few hundred bins. If more than 1000 or so bins are used, the program will become very slow and it may be quicker to use vtwspec.


vtwspec

A wideband spectrum analyser.
   vtwspec [options] input

options:
 -B            Run in background
 -L name       Specify logfile
 -r hertz      Resolution (bin width)
 -a            Non-coherent averaging until end of input
 -N frames     Average up to this many transform frames, then exit

 -W window     Specify window function:
                   rect (default), cosine, blackman, hamming,
                   nuttall, hann

examples:
   # Average 60 seconds of source in 1Hz resolution
   vtcat -E60 source | vtwspec -a -r1 > datafile

All channels of the input stream are transformed to the frequency domain using a FFT and the resulting spectrum is sent to the stdout. Unless the -a option is given, the program does a single transform of length sample_rate/resolution and then exits. If -a is given, the program performs successive FFTs until the input stream ends, averaging the signal power in each bin. A -N option can be given to tell vtwspec to end the averaging and terminate after the given number of FFT frames.

The transform output is sent to stdout in ASCII format with one row per frequency bin. The first column is the bin frequency in Hz. Without -a there are three further columns for each channel: real amplitude, imaginary amplitude, and the RMS magnitude. With -a, there is a single column per channel: the RMS magnitude.


vtpolar

Polar A/V display of orthogonal signals.
   vtpolar [options] [input [output]] 

options:
 -v            Increase verbosity
 -B            Run in background
 -L name       Specify logfile
 -s size       Display diameter, pixels (default 224)
 -r rate       Frame rate (default 10)

 -m mode       Display mode (default tf)
                 -mf   spectrogram
                 -mt   time domain (vectorscope)
                 -mtf  both

 -b size       Buffer size before ffmpeg (default 10)
 -k from,to    Frequency range, Hz (default DC to Nyquist)
 -p polarspec  Specify alignment of input channels
 -c            Paint background

 -ga=gain      Audio gain (default a=1.0)
 -gv=gain      Video gain (default v=1.0)

 -f options    Options passed to ffmpeg
    options is a comma separated list of parameters:
     format=container_format, (default avi)
     vcodec=video_codec, (default mpeg4)
     acodec=audio_codec, (default libmp3lame)
     vbr=video_bitrate, (default 100)
     abr=audio_bitrate, (default 64)
examples:
   # Pipe mpeg A/V stream to mplayer
   vtpolar -f format=mpegts,vbr=1000,abr=128 -s400 -p96,6 @hloops | mplayer -

   # Produce an avi file from 1 hour of signal extracted from saved data
   vtread -T2010-10-15_05:00,+3600 /data | vtpolar -s360 -r10 - file.avi

   # Send an A/V stream to ffserver.   The ffserver config specifies the
   # bitrates, container and codecs.  We specify the size and frame rate.
   vtpolar -s400 -r12 @hloops http://server/stream.ffm

This program takes a 2-channel stream from a pair of loops, or 3-channels consisting of 2 loops and an E-field signal, and produces a polar display of frequency and bearing and/or amplitude and bearing. Bearings are extracted mod 360 if an E-field channel is available, otherwise they are mod 180.

The display video is multiplexed with the audio stream and encoded using ffmpeg. output is a pipe, filename, or the uplink URL to an ffserver. Use ffmpeg -formats to get a list of available container formats and codecs. Some container formats, such as swf will only accept certain audio sample rates, in which case use vtresample before vtpolar.

The -p polarspec option indicates the assignment and alignment of each of the input channels. For example -p 354,93,E would indicate a three channel input with channel 1 from a loop oriented on 354 degrees, channel 2 oriented 93 degrees, and channel 3 is an E-field channel. For good results it is essential to equalise the phase response of the receivers involved. The loop signals do not have to be orthogonal.

A 10 second buffer is maintained between vtpolar and ffmpeg. The size may be changed with the -b seconds option but if the buffer is too small a deadlock will occur and vtpolar will go into a busy loop.

At present, the http uplink option seems to be broken due to a problem with ffmpeg or ffserver.


vtplot

Time domain plotting
   vtplot [options] [input] > imagefile

options:
 -s x,y      Plot size in pixels, default 640,480
 -t title    Title for the plot
 -o format   Specify output image format
examples:
   # Show available output image file formats
   vtplot -o?

   # Plot 15mS of data extracted from a signal database
   vtread -T2011-04-12_03:27.7246,+15e-3 /datadir |
     vtplot -s 1200,600 -t "Nice multimode tweek" -o x11

   # Plot 30mS from the real-time data in @source
   vtcat -E0.03 @source | vtplot -t "Live snapshot" -o 'png small' > live.png

vtplot is a shell script which uses vtraw -oa to convert the input stream to ASCII data and then invokes gnuplot to do the plotting.

If no -o option is given, vtplot will display the time domain waveform on the X display. Enter 'q' into the persistent display window to remove it.


vtsidex

Extract records from the data files produced by vtsid.
   vtsidex [options] datadir

options:
 -v            Increase verbosity
 -d datadir    Specify data file root directory, no default
 -m monitor    Specify a monitor.  Defaults to spectrum data.
 -o format     Specify output format options, see below
 -T timespec   Restrict records to a range of times.  Default is all records
 -F freqspec   Restrict spectrum columns to a range of frequencies. Default is
               all frequencies.
 -f filename   Specify a particular file name
 -s            Report status
 -a n          Average over n raw records
 -h degrees    Apply hysteresis to angle values
 -t            Tail the data

Selected records are extracted from the vtsid data files under the directory given by the datadir. If a -m option is given, the monitor's records are extracted, otherwise spectrum data is extracted.

Option -T timespec delimits the time range to be extracted. See Time specifiers for a description of timespec.

When extracting spectrum data, option -F freqspec can be used to restrict the frequency range extracted. freqspec takes the form start,end where start and end are frequencies in Hertz. Either can be omitted and start defaults to zero and end defaults to the Nyquist frequency.

Output options are introduced by -o flags. These are:

 -ote   Timestamps in unix epoch, seconds since 1970-01-01 00:00:00
 -oti   Timestamps in format YYYY-MM-DD_HH:MM:SS.SSSS  (default)
 -otr   Timestamps in seconds offset from the start time
 -oh    Output a heading record at the start of data
 -ohn   Issue a heading record every 'n' data records
 -otb   Output a blank line when there is a timing gap in the data
The -otb option is useful when piping the data into gnuplot, causing the timing break to be represented by a gap in the chart.

A summary of a data file is produced by the -s option. A filename may be given, in the form YYMMDD-HHMMSS, otherwise the current data file is reported. The -m option specifies which monitor is referred to.

Records are averaged if -a n is given. An output record is generated after every n raw records. The timestamp assigned is the timestamp of the first raw record of the average. If a timing break occurs in the input stream, a new average is started. Bearings and phase angles are averaged correctly, modulo 180 or 360 according to the type of raw data field.

To avoid too much wrapping of phase and bearing measurements, a hysteresis can be applied with -h degrees. This is useful for plotting noisy angles that slowly move through the wrapping angle. For example -h20 will allow a 0-360 degree bearing to range from -20 to 380 degrees, and a -180 to +180 phase angle will use the range -200 to +200. This option can also be used with a large hysteresis, eg -h5000 to completely unwrap long phase changes, such as occur during the day/night transitions during which the signal phase can change by several complete cycles.


vtsidplot

Plot data from vtsid monitor data
   vtsidplot -m monitor -T timespec [options] datadir [> imagefile]

options:
 -m monitor     Specify a monitor
 -T timespec    Time range
 -s x,y         Plot size in pixels, default 640,480
 -t title       Title for the spectrogram
 -a count       Average over count raw records
 -h degrees     Apply degrees of hysteresis to phases and bearing

 -f fields      Specify fields to be plotted (default all)
                Use -f? to show available fields

 -o format   Specify output image format (default x11)
             vtsidplot -o? for available formats

examples:
   # List the available fields for monitor NAA
   vtsidplot -m NAA -T2011-03-18 -f? /raw/sid

   # Plot carrier phase and bearing for a whole day, into a png file
   vtsidplot -m NAA -T2011-03-18,+86400 -f 'cp1 b360' -o 'png small' /raw/sid > out.png

This is a script which invokes vtsidex to extract the requested data and runs gnuplot to do the plotting. The options -m, -T, -a, -h and the database directory are passed down to vtsidex. See the description of vtsidex for their usage.

If -f fields is given then only the requested fields are plotted. The fields argument is a comma separated, or a quoted space separated, list of fields. A -f? option will report the available fields and a -T option must be given with this because the available fields may depend on the configuration of vtsid at that time.

Due to a limitation of gnuplot, this utility is only useful when the monitor logging interval is greater than one second. If not, then vtsidplot may still be used if a -a count option is given such that the interval between averaged records is greater than one second.


vtsidgram

Plot spectrograms from vtsid spectrum data
   vtsidgram [options] datadir [> imagefile]

options:
 -s x,y      Plot size in pixels, default 640,480
 -t title    Title for the plot
 -o format   Specify output image format
 -T timespec Specify time range
 -F freqspec Specify frequency range
examples:
   # Produce output in png format and display with 'xv'
   vtsidgram -T2011-10-17_10:30,+3600 -F12500,14500 -o 'png small' -s320,480 /raw/sid | xv -

This is a script which runs vtsidex to extract spectrum data and passes it to gnuplot to render in pm3d mode. The -T timespec and -F freqspec options are passed on to vtsidex and are described above. With no -o option, the x11 display is used, otherwise an image file is sent to stdout.

vtsidgram selects the amplitude fields from the output of vtsidex and if there is more than one amplitude field (multiple channels), the RMS sum of the amplitudes is used.


  Utility Programs

A selection of programs useful for monitoring the signal processing modules.

vtps

Display all VT processes.

   vtps [options] [modes]

options:
 -f          Full format
 -F          Extra full format
 -l          Long format
 -o format   Custom format options

modes:
   u       Show memory and cpu used
   v       Show virtual memory
   w       Wide listing

This program is just a front-end for the ps utility, which is run for every VT process on the host. The options and mode letters on the vtps command line are just passed on to ps. See the ps man page for more options.

If no options are given, ps is run with the options -o pid,rtprio:2=RT,pcpu,pmem,command.


vttop

Active display of VT processes.

   vttop [options]

options:
 -b          Run in batch mode (useful for scripting)
 -d ss.tt    Delay between updates, seconds.tenths

This program is a front-end for the top utility, running top for all VT processes on the host. Options are just passed straight on to top. See the top man page for more options and interactive commands.


vtwait

Wait for data on a stream

   vtwait [options] input

options:
 -v          Increase verbosity
 -t          Wait for data on the stream
             (Default is to wait for buffer creation
examples:
   # Run vtcard to read the soundcard into buffer @raw
   vtcard -B -d hw:0,0 -r 48000 @raw,20,i2

   # Wait for @raw to be ready
   echo "waiting for @raw..."
   vtwait -t @raw || {
      echo "buffer @raw invalid" >&2 
      exit 1
   }

   # Start programs which use @raw for input
   vtfilter ... @raw -
   vtwrite ... @raw

This program is intended for use in start-up scripts in which vtcard or vtvorbis is used in background to run data into a lock-free buffer. The program waits for the buffer to be created, and if -t is given, it continues to wait until data is available.

Without this program, the start-up script would proceed to launch subsequent processing modules which may fail immediately because their input buffer does not yet exist.


vtcardplot

Plot soundcard performance

   vtcardplot [options] vtcard_logfile [> imagefile]

options:
 -d date     Plot only the specified day, yyyy-mm-dd (default whole logfile)
 -s x,y      Image size (default 640,480)
 -o format   Output format (default x11)
examples:
   # Plot the performance of the vtcard and display on x11
   vtcardplot -d 2011-11-18 /tmp/vtcard.loop_rx.log

   # Show available output formats
   vtcardplot -o?

This is a script which uses gnuplot to graph the soundcard conversion rate and timestamp error logged by a vtcard program. The vtcard logfile must be given and if no -d date option is supplied, the entire log file is plotted.

The offset graph shows the timing offset of the output stream relative to the system clock. A positive offset indicates that the stream timestamp is late (has a higher timestamp value).


vtdate

Conversion and operations on timestamps

   vtdate [options] [timestamp|timespec]

options:
 -v       Increase verbosity
 -i       Integer output
 -n       Output numeric timestamps
 -s       Output ISO format timestamps

 -t secs  Truncate to a multiple of secs seconds
 -a secs  Add secs seconds to the timestamps 
examples:
   # Get the current time as an ISO string
   vtdate -s now

   # Split a timespec into numeric start and end fields
   set -- `vtdate -n 2011-10-09_08:32:17.49,+2400`

   # Subtract an hour from a timestamp
   vtdate -a -3600 2011-10-09_08:32:17.49

   # Convert a numeric timestamp to ISO form
   vtdate -s 1318151537.49

This utility takes a timestamp or a timespec and prints their values to stdout in accordance with the -n and -s options. Outputs have a fixed length canonical form, eg

   1318145537.490000
   2011-10-09_07:32:17.490000
If neither -n nor -s are given, both forms are output. Output fields are space separated and if a timespec is given, the start and end times are output in separate fields. A -i option will discard the decimal point and following digits.

This utility is intended for use in scripts that need to manipulate timestamps.


vtspot

Determine source locations from arrival times and bearings.

   vtspot [options] meas1 [meas2 ...]
   vtspot -b standpoint forepoint
   vtspot -d standpoint bearing range

options:
 -c factor    Specify a velocity factor (default 0.9922)
examples:
   # Bearing and range of 44.6N,67.3W as seen from 53.7N,2.1W
   vtspot -b 53.7N,2.1W 44.6N,67.3W

   # Determine source from two arrival times and a bearing
   vtspot T/39.9N,74.9W/1334382837.867327 T/53.7N,2.1W/1334382837.884484 B/53.7N,2.1W/298

This program takes an arbitrary mix of arrival time and bearing measurements and calculates the most likely source locations and times for the signal. All the calculations are performed using n-vectors to avoid singularities at the poles.

Measurements may be given on the command line or in the standard input stream. Each measurement must conform to one of the following formats:

   T/location/time[/std_dev] 
   B/location/bearing[/std_dev]
   A/location1/location2/atd[/std_dev]
   I/ident
'T' measurements specify an arrival time, 'B' measurements specify a bearing, and 'A' records specify an arrival time difference (for example from a cross-correlation). Each measurement can specify a standard deviation. Arrival times are given as a timespec, see Time specifiers for the description of timespec. Bearings are given in degrees mod 360. Standard deviations are specified in seconds for timestamps and ATDs, and degrees for bearings. These default to 100e-6 seconds and 5 degrees. Locations are specified as latitude,longitude and the following examples illustrate the formats recognised:
   53.703N,2.072W
   53.703,-2.072
   53:42:10.8N,2:04:19.2W

The collection of measurements is referred to as a 'measurement set'. A measurement set must contain at least two bearings, or three arrival times, or two arrival time differences. Combinations are allowed, such as two arrival times and one bearing, or one ATD and a bearing. If the measurement set is given on the command line, vtspot processes the measurement set and then exits. Multiple measurement sets can be presented to vtspot on the standard input stream with one measurement set per line and each will be processed independently.

An 'I' field in the measurement set tags the set with an identifying string token which appears in the output records. This is useful when reading from standard input in order to associate each result with its measurement set.

If a solution cannot be found for a measurement set because the measurements do not intersect anywhere (for example one or more of the timestamps or bearings are not consistent with the others) then no output will be made for that measurement set.

Some measurement sets will have two solutions, for example a pair of bearings or three arrival times will have two exact solutions. Two solutions can also occur with some measurement sets where the receivers are not well located with respect to the source.

Results go to standard output with the following format:

   ident index latitude,longitude timestamp score
The ident field will be missing if the measurement set did not supply a I/ident field. Index is 0 or 1 to distinguish the two solutions possible with some measurement sets. The score is the worst error of the solution tested against each measurement of the set and is in units of standard deviations. An example output looks like
   test1 0 30.867,-75.571 1334382837.863869 4.216e+00
   test1 1 35.856,-99.912 1334382837.859827 2.143e-03

Locations can be given symbolically using site names defined in a file named spots. An example file is included in the package. This should be placed in the working directory or in the user's home directory. The symbolic names can be used wherever the program expects a latitude,longitude.

A -b option computes the bearing of forepoint as seen from standpoint. A -d option computes the forepoint given a standpoint, bearing and range in km.


  Soundcards

This package relies on ALSA or OSS drivers to deal with the soundcard. The program vtcard is used to read data from the soundcard. For each soundcard, a vtcard process must be started to transfer data from the card to a lock-free buffer.

Begin by running vtcard with -vv which will report some information about the soundcard. For example,

   vtcard -vv -r96000 -d hw:0,0 @test
You may get an error such as "cannot set channel count" if the card does not have two channels, in which case use option -c1. The output will look something like
 vtcard: buffer name: [@test]
 vtcard: using ALSA device hw:0,0
 vtcard: rate min 44100 max 96000
 vtcard: sample rate set: 96000
 vtcard: period size: 256 frames, 2666 uS
 vtcard: buffer size: 8192 frames, 85333 uS
 vtcard: periods per buffer: 32
 vtcard: nread: 256
 vtcard: avail min: 256
 vtcard: channels: 2  bytes: 2
 vtcard: using SCHED_FIFO priority 99
 vtcard: pre-run complete 96011.35
 vtcard: setup -2.612e-04 sr 96007.59 rr 95986.3 n=0 r=0.86
 vtcard: setup +3.878e-05 sr 96008.15 rr 96011.3 n=1 r=0.13
 vtcard: setup +3.024e-04 sr 96012.50 rr 96037.2 n=3 r=0.99
 vtcard: setup -2.767e-04 sr 96008.52 rr 95985.9 n=4 r=0.91
 vtcard: setup +2.870e-05 sr 96008.93 rr 96011.3 n=5 r=0.09
 vtcard: setup +2.933e-04 sr 96013.15 rr 96037.1 n=8 r=0.96
 vtcard: setup -2.850e-04 sr 96009.05 rr 95985.8 n=9 r=0.93
 vtcard: run +2.715e-05 sr 96009.180 tadj +2.320e-06 rr 96011.7 offs +0.271mS r=0.09
 vtcard: run +2.122e-05 sr 96009.282 tadj +1.814e-06 rr 96011.2 offs +0.212mS r=0.07
 vtcard: run +2.246e-05 sr 96009.390 tadj +1.920e-06 rr 96011.4 offs +0.225mS r=0.07
 vtcard: run +2.070e-05 sr 96009.489 tadj +1.769e-06 rr 96011.4 offs +0.207mS r=0.07
   ...
The program runs in 'setup' state until it is happy with the smooth flow of data and then switches to 'run' state. During 'setup' it looks for r= values less than one. If the r= value does not settle down the card will not go into 'run' state. In this event, note the default value of period size and use a -Ap= option to try different sizes. Stick to powers of two and try values 2 or 4 times the default, or 1/2 or 1/4 of the default.

The buffer size is not important because vtcard should be scheduled promptly to read from the soundcard driver whenever a single period is available.

Once the program enters 'run' state, it will deliver data into the output stream, in this case buffer @test. It will take about 2 minutes to do the setup. Monitor the signal level in the buffer with

   vtstat @test
which will report the peak value. Adjust mixer controls so that the VLF signal is peaking around 0.2 to 0.5 which leaves a small amount of headroom.

If at full mixer gain you are unable to reach a reasonable peak signal, that indicates you need more analogue gain before the line input. If this is not available then you can use a -g option to apply a software gain to the soundcard samples - at the expense of dynamic range. For example, if vtstat is reporting a peak of 0.1 at best, then you might restart vtcard with a -g4 option.

If the card can do 24 bit conversion, you will probably need to use option -b32. The card returns 24 bit samples in a 32 bit words with the lowest significant byte set to zero.

If the soundcard conversion rate changes too fast for vtcard to deal with, or if the system clock makes a step change or slews too fast, vtcard will drop back into 'setup' state. Output will stop until the 'run' state is regained and processes taking data from the output buffer will detect a timing break when the stream restarts. To check for this you can leave vtstat @test running and it counts up the number of breaks detected.

When satisfactory operation is obtained, restart vtcard with a -B option to background it, and a -L option to create a log file.

When vtcard has been running for a few minutes, the offs value will be giving small random positive and negative values, for example

2011/11/18 16:43:57 run +2.345e-06 sr 96011.471 tadj +2.004e-07 rr 96011.7 offs +0.023mS r=0.01
2011/11/18 16:44:07 run +4.275e-07 sr 96011.473 tadj +3.655e-08 rr 96011.5 offs +0.004mS r=0.00
2011/11/18 16:44:17 run -1.159e-06 sr 96011.467 tadj -9.938e-08 rr 96011.4 offs -0.012mS r=0.00
2011/11/18 16:44:27 run -1.901e-06 sr 96011.458 tadj -1.625e-07 rr 96011.3 offs -0.019mS r=0.01
2011/11/18 16:44:37 run -1.906e-06 sr 96011.449 tadj -1.629e-07 rr 96011.3 offs -0.019mS r=0.01
2011/11/18 16:44:47 run +5.188e-06 sr 96011.474 tadj +4.435e-07 rr 96011.9 offs +0.052mS r=0.02
2011/11/18 16:44:57 run -1.051e-07 sr 96011.473 tadj -1.278e-08 rr 96011.5 offs -0.001mS r=0.00
2011/11/18 16:45:07 run -1.430e-05 sr 96011.405 tadj -1.222e-06 rr 96010.1 offs -0.143mS r=0.05
2011/11/18 16:45:17 run +1.212e-05 sr 96011.463 tadj +1.035e-06 rr 96012.6 offs +0.121mS r=0.04
2011/11/18 16:45:27 run +1.944e-06 sr 96011.472 tadj +1.662e-07 rr 96011.6 offs +0.019mS r=0.01
2011/11/18 16:45:37 run -4.188e-06 sr 96011.452 tadj -3.580e-07 rr 96011.1 offs -0.042mS r=0.01
2011/11/18 16:45:47 run +9.563e-07 sr 96011.457 tadj +8.483e-08 rr 96011.5 offs +0.010mS r=0.00
2011/11/18 16:45:57 run -1.591e-06 sr 96011.449 tadj -1.360e-07 rr 96011.3 offs -0.016mS r=0.01
2011/11/18 16:46:07 run +2.988e-06 sr 96011.463 tadj +2.554e-07 rr 96011.7 offs +0.030mS r=0.01
2011/11/18 16:46:17 run -1.796e-06 sr 96011.455 tadj -1.535e-07 rr 96011.3 offs -0.018mS r=0.01
offs is the residual error of the timestamped stream with respect to the system clock. sr is the smoothed sample rate, rr is the raw sample rate taken oven the previous 10 seconds.

You can use the script vtcardplot to graph the performance of the soundcard and timestamping. The script scans the logfile of vtcard to extract the sample rate and timing offset columns. You must specify the full pathname to the logfile, eg

   vtcardplot -s800,400 /tmp/vtcard.test.log
produces a plot size 800 by 400 on the x11 display. A date can be given with a -d option and vtcardplot will report just that date. Otherwise it reports the whole logfile. Alternate output formats can be request with -o option.
   vtcardplot -d 2011-11-18 -o 'png small' /tmp/vtcard.test.log > out.png
The graphs reveal the variation of souncard conversion rate. vtcard tracks these variations and the current sample rate is reported in the output stream in the sample rate calibration factor. If the conversion rate changes too quickly, vtcard will reset to setup state to reestablish timing. This should be avoided if possible. In one case, resets occured on windy days and the problem was traced to a draught blowing in through the air vents on an outdoor equipment cabinet and being drawn into the PC air intake. The cure in that case was to glue some foam over the soundcard's conversion rate crystal to increase the thermal time constant.

In running state, vtcard applies two servo loops, one to slew the output timestamp error towards zero, and the other to track the soundcard conversion rate. In setup state only the sample rate servo is active and this must settle first before the program will enter run state and enable the timestamp correction loop.


  Timestamps and alignment

The program vtcard assigns timestamps to stream packets by reference to the system clock, which should be disciplined by ntpd. The resulting timestamps have an accuracy of about +/- 50mS. This is adequate for some applications, for example SID monitoring and casual whistler detection. Other applications may require more accuracy, for example:

For these applications, the program vttime will refine the timestamp by reference to timing signals within the streams. A timestamp accuracy of better than 1uS can be achieved when vttime is given a GPS PPS reference.


Using a pulse-per-second

Example PPS on the 2nd channel with VLF on channel 1.
A channel of the soundcard must be dedicated to receiving a PPS signal. Typically a single channel of VLF will occupy channel 1 of the soundcard and the PPS will be on channel 2. The pulse waveform needs to have moderately slow rise and fall times and should not have a flat (clipped) top. A simple RC low pass network can be used to convert the rectangular pulse from a GPS into a suitable shape. In the example on the right, the original rectangular PPS is about 0.9mS wide and begins on the second mark.

vttime works by looking for the centroid of the pulse, which it can do very accurately if the PPS input is low noise (no soundcard hiss or spurious impulses) and the edges of the pulse rise and fall smoothly to a well defined peak with no flat top. The pulse amplitude should use most of the dynamic range of the soundcard.

The leading edge of the PPS normally represents the timing mark, so vttime must be told via the c= mode option where the centroid sits in relation to the timing mark. This is something that has to be calibrated. It is desirable for the time constant of the RC shaping network to have low temperature dependence or be kept at a fairly constant temperature, as this affects the delay between leading edge and centroid and therefore directly affects the timing calibration.

For maximum noise immunity, vttime first looks for the pulse peak and then restricts its integration of the centroid to a symmetrical period of time either side of the peak. This half-width is given by the w= mode parameter. It should be set to a little larger than the rise/fall time of the pulse to guarantee that the whole pulse is integrated.

The input stream to vttime must be reasonably well timestamped by vtcard which is ensured by running ntpd on its host. vttime uses the incoming timestamp to get a rough idea of when to look for the pulse. It is convenient to use gpsd as a timing source for the ntpd.


Timing system setup

Assume that the raw soundcard signal is available in a buffer @raw, with VLF on channel 1 and a PPS on channel 2. Then test vttime by running

   vttime -vv -m ppsbase+,c=0,w=2e-3 -c2 @raw @timed
The w= value should be set to a little over half the overall width of the shaped pulse. To begin with, err on the larger side. The output will look something like
 vttime: selected channel: 1 = @raw:1
 vttime: selected channel: 2 = @raw:2
 vttime: channels: 2, input rate: 192000
 vttime: centroid offset: 0.000e+00
 vttime: buffer size: 703 blocks, 29.995 seconds
 vttime: break detected on ++9802: 79.869
 vttime: wild PPS pulse interval 19829.855520 - skipped
 vttime: st0 PPSpmr 164.4 PPSsig 0.000uS in 22.358mS out   0.000uS rate_err  3.38 inrate 192001.6895 int 192003.3791
 vttime: inrate 192002.6570 int 192003.3424
 vttime: st0 PPSpmr 164.5 PPSsig 0.041uS in 22.259mS out   6.639uS rate_err  0.71 inrate 192003.0122 int 192003.3674
 vttime: st0 PPSpmr 164.6 PPSsig 0.055uS in 22.227mS out   3.549uS rate_err  0.40 inrate 192003.2118 int 192003.4114
 vttime: st0 PPSpmr 164.3 PPSsig 0.055uS in 22.194mS out   1.537uS rate_err  0.15 inrate 192003.2889 int 192003.3659
 vttime: st2 PPSpmr 164.0 PPSsig 0.055uS in 22.162mS out   0.778uS rate_err  0.08 inrate 192003.3020 int 192003.3678
 vttime: st2 PPSpmr 164.1 PPSsig 0.054uS in 22.130mS out   0.949uS rate_err  0.07 inrate 192003.3138 int 192003.3729
 vttime: st2 PPSpmr 164.3 PPSsig 0.068uS in 22.097mS out   0.859uS rate_err  0.02 inrate 192003.3180 int 192003.3387
 vttime: st2 PPSpmr 164.1 PPSsig 0.076uS in 22.066mS out   1.173uS rate_err  0.09 inrate 192003.3333 int 192003.4100
 vttime: st2 PPSpmr 163.9 PPSsig 0.074uS in 22.034mS out   1.159uS rate_err  0.05 inrate 192003.3417 int 192003.3834
 vttime: st2 PPSpmr 164.0 PPSsig 0.074uS in 22.002mS out   1.186uS rate_err  0.05 inrate 192003.3501 int 192003.3924
The program spends a little time in state st0 which is a setup state, then switches to st2 which is the normal running state. The output reports a value PPSsig which is the standard deviation of the centroid location. This should be quite a bit less than a uS if everything is working. PPSpmr is the peak/mean ratio of the PPS channel and should be 50 or 100 or more.

The next step is to set the w= parameter more carefully. Use vtscope to estimate the overall width of the pulse on channel 2 and add 5 or 10%, then restart vttime using half this value. You can also set an approximate value for the c= parameter at this point, setting it equal to the shaped pulse half-width.

Allow vttime to run a while in foreground to see that it is stable, then restart in background, giving it a -B option and specify a -L logfile.

The output channel @timed contains the VLF signal on channel 1 and the timing pulse on channel 2. You can use vtscope or vtplot to see that the timing pulse now begins to rise exactly on or very close to the second mark of the stream's timestamp.

For accurate frequency and phase measurement, the above is sufficient. If timestamp accuracy is required, then the centroid offset needs to be calibrated. This was approximately set above to the half width of the shaped PPS pulse. The setting can be improved by disconnecting the VLF and feeding the raw rectangular PPS pulse into channel 1 of the soundcard via a small capacitor so that a short sharp pulse, just 1 or 2 samples wide, is captured. There will be two such pulses, one for the leading edge and another for the trailing edge of the PPS. Use vtscope or vtplot, or examine the ASCII data produced by vtraw -oa to measure the timestamp at which the the leading edge pulse on channel 1 begins to rise. This will probably be slightly offset from the stream timestamp's second mark. Adjust the c= value to correct for this. If the leading edge pulse starts early with respect to the stream timestamp, this means the stream timestamp is late with respect to UT and the centroid offset needs to be increased by that amount.

With the method described above, the centroid offset can be calibrated to about the nearest sample. Further improvement requires some ingenuity.


  RTL2832U


Installation

You may need to install librtlsdr. Download from https://codeload.github.com/steve-m/librtlsdr/tar.gz/v0.5.2 and build with:

   tar xzf librtlsdr-0.5.2.tar.gz
   cd librtlsdr-0.5.2
   mkdir build
   cd build
   cmake ../
   make install
   ldconfig 

You may also need to install libusb development files. On Ubuntu, do
   apt-get install libusb-1.0-0-dev 

To include vtrtlsdr, you must configure with the option --with-rtlsdr.

If you find that

   vtrtlsdr ... | vtmix -c1,-j
gives you lower sideband (inverted spectrum) instead of upper sideband as you would expect, then use vtrtlsdr -q to invert the Q signal to put things right.

Annoyingly, librtlsdr throws lots of gratuitous messages to stderr during setup. You could redirect stderr but then you lose your own log messages unless you're using -L. Determined users will hack into the librtlsdr source to clean it up.


  Hints and Tips


Playing a stream

Use vtraw -ow to convert the stream to WAV format and pass this onto aplay or whatever raw PCM player you use. You may want to resample the stream first, filter it, or change the number of channels. For example, to play the 2-channel stream @source at a 44100 sample rate, you might do

   vtresample -r44100 -q1 @source | vtraw -ow | aplay -
You can produce mp3 using lame. For a mono stream at 64kbps,
   vtresample -r44100 -q1 @source | vtraw -ow | lame -b64 -mm - - > output.mp3
To produce a vorbis file or stream, you don't need to convert to raw first, just pipe the data through vtvorbis -ep,
   vtresample -r44100 -q1 @source | vtvorbis -ep
You can configure xinetd to run any of these pipelines in response to a network connection, making an easy way to distribute a playable stream on your LAN.


Whistler detection

This is very straightforward because vtevent requires little or no configuration and automatically adjusts itself to the characteristics of the input stream. It is desirable to reduce the sample rate to 32k/sec or 48k/sec and filter the hum first before presenting the signal to vtevent. For example, with a stream @raw containing a single E-field or loop signal, the following might be used,

   vtresample -r32000 @raw | vtfilter -ath=6 | vtevent -v -d outdir

If orthogonal loops are available, whistler bearing and polarisation can be extracted if the program is given a -p option. For this to work properly, the two or three inputs must be calibrated and corrected for phase response using vtfilter -e eqmap. In 3-axis mode of operation, the whistler data shown here has been compiled.

A reassigned spectrogram produced by vtevent brings out the tendrils of triggered emission caused by this whistler.

Detected whistlers are captured and saved under the directory outdir. For each event, the program records a thumbnail spectrogram, a 6 second stream file, and a text file containing information about the event. Examples are found in the event pages under here.

The event detector applies principal component analysis to the short-term Fourier representation of the signal to identify regions of the F,T space that contain curvilinear features. A Hough transform then recognises when several regions combine to conform to a whistler curve. Risers are detected by the presence of sufficient regions containing rising features. The detector is not very sensitive to diffuse activity such as hiss bands or diffuse chorus.

The program normalises the input signal by looking for the VLF noise floor in between the sferics and then the signal is equalised and the dynamic range compressed with respect to the this.


Weak signal detection

When the received signal is accurately timestamped by GPS using vttime, it is suitable for extraction of weak narrow-band signals such as used by radio amateurs. During reception the signal should be passed through vttime and into raw storage via vtwrite. Signal detection is then carried out during post-processing using vtread to extract the raw data.

The tool to use for narrow band detection is vtnspec. It is essential to blank the sferics from the signal before passing it to vtnspec and in turn, to operate properly, the blanker vtblank must have its input stream band-limited to a width of 2 or 3kHz. Prior to band-pass filtering, any beam synthesis and steering should be performed by vtmix on the raw signal extracted by vtread.

For example, to look for a signal at 8970.00245Hz which was transmitted for one hour starting 2011-08-12 06:15 UT, you might use something like the following pipeline,

   vtread -T2011-08-12_06:15,+3600 /rawdir |
     vtmix -c 0.643,-0.766 |
       vtfilter -h bp,f=8970,w=3000 | 
         vtblank -a1.2 -d0 -t1 |
           vtnspec -f 8970.00245 -w0.040 -r278e-6 > spectrum.dat
vtread extracts 3600 seconds of signal from orthogonal loops stored in /rawdir and the vtmix produces a single output channel synthesising a loop oriented on the desired bearing. vtfilter limits the bandwith to +/-1.5kHz of the operating frequency and the options to vtblank are reasonably optimum for normal VLF.

vtnspec analyses the spectrum over the range 8970.00245Hz +/-20mHz using frequency bins of width 278uHz, which is 1/3600 seconds. The output file is ASCII with frequency in column 1 and RMS amplitude in column 4.

A 5uW ERP rubidium controlled carrier from DF6NM received at 1030km range.

A spectrum plot can be scripted with something like the following

   echo "set terminal png small
         set style data lines
         set xlabel 'Frequency - 8970Hz'
         set ylabel 'Relative amplitude'
         unset title
         plot 'spectrum.dat' using (\$1 - 8970):4
     " | gnuplot > spectrum.png
or you can just use your favourite plotting tool interactively. The example spectrum on the right is produced by commands very similar to above. This is an eight hour integration of a 5uW transmission from DF6NM at a range of 1030km. Column 4 has been turned into fT by multiplying by the system's calibration factor.

Storing the raw signal and then post-processing is strongly recommended, as it allows different antenna orientations, polarisations, and bandwidths to be tried.

If the signal is strong enough, columns 2 and 3 of the output of vtnspec can be used to report the absolute phase of the received signal. To do this, it is first necessary to establish the average frequency of the signal, then re-run the analysis using a vtnspec set for a single bin centered on the average frequency. Running a sequence of short, overlapping single bin 'spectra' allows the signal phase to be reported against time. This reveals the variations in path length and will identify glitches in the transmitter phase and gives some idea of the maximum symbol duration usable by differential phase shift keying.


Stream server

A simple stream server can be set up using vtcat and xinetd. Assume you have a buffer called @source on the server which you want to make available to other hosts on your LAN, on demand. Define a service with xinetd by creating a file /etc/xinetd.d/vt-source, containing the following

   service vt-source
   {
            type = UNLISTED
              id = vt-source
            port = 4420
     socket_type = stream
        protocol = tcp
            user = root
            wait = no
         disable = no
       only_from = 192.168.2.0
          server = /usr/local/bin/vtcat
     server_args = @source
   }
You might want to pick your own spare port and suitable user ID. Reconfigure xinetd with the command
   killall -HUP xinetd
Now, when anyone connects to port 4420 on the server, they will receive the output of the command
   /usr/local/bin/vtcat @source
Clients can then connect to and receive the stream using netcat, for example,
   nc server 4420 | vtstat
and so on. Of course, you can be more creative with the server command and arguments. For example you might prefer to stream vorbis encoded, in which case you might have
         server = /usr/local/bin/vtvorbis
    server_args = -e -q0.8 @source
in the xinetd service definition file.

Setting up a stream server doesn't get much simpler than this.


Storage and retrieval

It is strongly recommended to set up a signal database to store raw or raw and timed, VLF signal. Disk space is cheap these days and there is no difficulty keeping 10 days or so of raw signal. With this facility, you have several days in which to hear about interesting events such as whistlers, aurora, flares, GRBs, TLEs, meteor fireballs, earthquakes, satellite re-entries, amateur radio transmissions, and so on.

Create a directory or perhaps mount a separate filesystem, to hold the data and use vtwrite to build the data files. Store the data as raw as possible (unfiltered and unequalised) although if you are using a GPS timing system, it is better to pass the raw data through vttime first before storage. You may also want to apply vtresample if you decide not to store at the full sample rate of the soundcard.

Assuming you have one or more channels of data direct from the soundcard in a buffer called @raw, and a raw data filesystem mounted on /rawdata, then create daily data files with

   vtwrite -B -G86400 @raw /rawdata
The stored format will be the same (f8,i2,i4, etc) as that of the buffer @raw. Maybe your raw buffer is 32 bits and you prefer to store in 16 bits, which should be satisfactory providing the VLF signal levels are using most of the 16 bit dynamic range. Then you must pass the signal through vtcat to change the sample format, eg with
   vtcat -B @raw -- -,i2 | vtwrite -B -G86400 /rawdata
We are using -B options to everything here so that the programs run in background. vtwrite starts a new file every 86400 seconds, ie one day, beginning at midnight. It will also start a new file whenever it sees a timing break on the input stream.

It is most desirable to have the raw data storage on a different host to that running the soundcard(s). This prevents vtcard operation being compromised by heavy use of the retrieval program vtread. In this case, use a network connection to transfer the data. For example, on the machine called 'bar' hosting the soundcard and raw buffer, run

   vtcat -B @raw ++foo,9876,i2
and on the machine called 'foo' which is to run the data storage, use
   vtwrite -B -G86400 ++9876 /rawdata
The use of '++' here allows either machine to start up first after a reboot, and causes the connection to be re-established after a network interruption.

If you're running multiple receiver channels, it is desirable to join them into a single multi-channel stream before storage.

The data files created in /rawdata can be used directly as input into any of the VT programs, but in most cases portions of data will be selected by using vtread to extract a particular time range, which is much faster and more efficient. For example, suppose you hear that there was a well documented TLE event occurring at say 2011-08-14, 16:37:25.256, and you want to see if you have the associated sferic, then you might run

   vtread -T2011-08-14_16:37:25.240,+50e-3 /rawdata | vtraw -oa > data.txt
which extracts 50mS of raw signal into data.txt which can then be plotted using gnuplot. A more refined command would run the data through vtfilter to remove the hum first, and this requires the data extraction to begin 10 or 20 seconds earlier to allow the automatic notch filter time to dispose of the hum. A suitable pipeline would be like
   vtread -T2011-08-14_16:37,+30 /rawdata |
     vtfilter -ath=6 |
       vtcat -T2011-08-14_16:37:25.240,+50e-3 |
         vtraw -oa > data.txt
Here, vtread starts early and extracts 30 seconds of data which is hum filtered, and vtcat is used to select the exact time range to be plotted. Of course, if you're doing a lot of this, you will have the above pipeline and the plotting ready in a script. See Time domain plots for help with the plotting of short signals.


SID monitoring

The toolkit contains an advanced SID monitor vtsid which is capable of monitoring multiple signals and records amplitude, bearing, and absolute phase. It can also capture the full spectrum of the VLF band. Data compiled by vtsid is stored in a database and the data is queried using vtsidex.

The procedure is to combine all the received channels into a single stream, pass this through vtfilter to apply your phase equalisation map, and send it all into vtsid. For example

   vtfilter -e eqmap @source | vtsid -vc config
You will have assigned a directory for storage of monitor and spectrum data, and prepared a configuration file config to suit your site and available stations. The buffer @source is expected to contain your combined signals, with timestamps polished by vttime if phase information is to be extracted.

A template configuration file is included vtsid.conf, - copy and edit this. A complete and up-to-date list of VLF stations is maintained by Lionel Loudet, available here.

Convenient monitoring intervals are 5 seconds for monitored channels and 120 seconds for spectrum data. If you are looking for high speed events such as LEPs, it is advisable to run a separate copy of vtsid configured to monitor just the relevant frequencies at a higher rate, eg 20 samples per second. In this case, distinct data directories must be used for each vtsid.

When vtsid is configured with a monitor_interval of zero, it will output records as often as possible and the rate is determined by the resolution or bins configuration. To obtain rapid sampling you must sacrifice resolution. For example, configuring the resolution to 50Hz with

   monitor_interval 0
   resolution 50
produces an output record every 20mS. You will need to set options mod=90,fast against any MSK monitors when sampling at high rates.

The options -m ident -t can be used to look at the current output for the specified monitor, for example,

   vtsidex -mNAA -t -oh10 /siddata
produces an output like
   ts a1 a2 cp1 b180
   2011-11-14_09:16:22.3492 1.31e-03 2.31e-04 156.6 104.6
   2011-11-14_09:16:27.3839 1.31e-03 2.34e-04 156.0 104.7
   2011-11-14_09:16:32.4186 1.29e-03 2.27e-04 165.1 104.5
   2011-11-14_09:16:37.4533 1.32e-03 2.30e-04 152.9 104.6
   2011-11-14_09:16:42.4879 1.31e-03 2.29e-04 158.7 104.6
   2011-11-14_09:16:47.5226 1.32e-03 2.32e-04 154.6 104.6
   ...
which continues until you terminate the program. The -oh10 option produces a heading every 10 records. a1,a2 are the RMS channel amplitudes averaged over the 5 second monitor inverval. cp1 is the MSK carrier phase (mod 180) and b180 is the apparent bearing of the incident signal. The columns reported will depend on the configuration of vtsid. In this example, vtsid is being fed with signals from a pair of loop antennas so there are two amplitude columns and a bearing column, and the 'phase' keyword is included in the config file so the program issues a phase column. If an E-field signal is also supplied to vtsid, it will report three amplitude columns and a 360 degree bearing column b360.

A similar vtsidex command is used to extract historical data, with a -T option specifying the time range,

   vtsidex -mNAA -T2011-10-06_23:30,+3600 -ote /siddata > data.txt
Here, a -ote option causes unix epoch timestamps to be produced which is convenient with the '%s' time format specifier in gnuplot. To plot the fluctuating phase and apparent bearing of NAA, you might then use gnuplot commands like
   set xdata time
   set time format '%s'
   set format x '%H:%M'
   plot 'data.txt' using 1:4 title 'phase', '' using 1:5 title 'bearing'

The -t and -T options can be combined, for example

   vtsidex -mNAA -Ttoday, -t -oh20 /raw/sid
reports everything from the previous midnight and then continues to follow the incoming records. Note the comma after 'today' which indicates there is no closing time.

The package includes a shell script vtsidplot which runs vtsidex and plots its output data using gnuplot. See vtsidplot for details. You can copy this script to a new name and customise to your local requirements. For example you can modify the scripts to factor in your receiver calibrations to plot vertical axes directly in pT or uV.

If you have several monitors and you want to plot them all together, you can make a simple script to run vtsidplot -o png on each monitor, all with the same -T timespec and then use the montage program to produce a composite plot.

Note that gnuplot cannot handle fractional timestamps. Therefore when there is more than one sample per second, you must make your own arrangements for plotting. gnuplot can still be used but the timestamp from vtsidex -ote must be treated as a numeric field. One solution is to subtract the previous midnight from the timestamp and plot the remainder as a 'Seconds since midnight' x-axis. Alternatively, subtract the first timestamp from all the timestamps so that the x-axis becomes an offset in seconds. The option -otr tells vtsidex to do this.

If vtsid is configured to log spectrum data, you can produce spectrograms from the stored data. See vtsidgram and Spectrogram plots for details.


Time domain plots

A sferic received by a pair of loop antennas, rendered by vtplot.

The toolkit contains a script vtplot to do simple time domain plots of a signal stream. Simply run a short length of stream into vtplot to get a display on your X terminal,

   vtcat -E0.01 @source | vtplot
Make sure you have something in the pipeline which restricts the length of the stream to a short sample. You can add a label to the X window by using
   vtplot -o "x11 title 'My signal'"
The mouse acts as a cursor in the persistent display window. Type a 'q' into the window to remove it. Send the output to an image file with
   vtplot -o "png small" > image.png
To customise plotting for your own requirements, for example you might want to do something different with the time axis or layout, copy vtplot to a new name and edit as required.

If you prefer another plotting program, or want to use gnuplot interactively, just use vtraw -oa to run the data into an ASCII file and apply your favourite tools to that.


Spectrogram plots

A strong whistler set against a background of natural VLF hiss.

A simple script vtsgram is included which produces spectrograms of short (up to a few minutes) time periods. It is a front-end for the Sox utility and assumes the installed version of Sox supports the spectrogram option. The example on the right was produced by the pipeline

   vtread -T2011-05-29_05:30:40,+60 /r3/raw |
      vtfilter -ath=6 -g4 -- -:1 |
         vtresample -q2 -r30000 |
            vtcat -T2011-05-29_05:30:53.3,+2.0 |
               vtsgram -p200 -b300 -s '-z60 -Z-30' > img840.png
A slightly longer than necessary section of raw data is extracted by vtread so that the filter can have 10 or so seconds of preamble in order to settle its automatic hum notching before the whistler comes along. The stream is resampled to 32k which sets the vertical scale of the spectrogram, and the -T option to vtcat is the one that selects the period of signal to plot.

The -s option allows extra options recognised by Sox to be passed along, in this case to set the z-axis range. These Sox options must be quoted into a single argument to vtsgram.

A second way to produce spectrograms is to use vtwspec or vtnspec to produce spectrum data and use the pm3d mode of gnuplot to do the plotting. This requires some scripting to run the spectrum analyser repeatedly, sliding an overlapping transform window along the required time axis. The resulting collection of spectrum data must be fed through awk to rearrange it for pm3d plotting mode. An example of this in operation is the real time spectrogram here.

A third method makes use of the spectrum data logged by vtsid. This is extracted by vtsidex for the required time range, rearranged by awk and plotted as above with gnuplot. This is well suited for quickly plotting long spectrograms spanning a day or several days. An example is shown below and the package contains a script vtsidgram for running these plots. Even longer time periods can be plotted this way by using synoptic sampling of the spectrum.

A 24 hour spectrogram, produced by vtsidex into gnuplot pm3d, shows a typical diurnal variation of sferic activity.

Multi-channel reception

The program vtjoin brings together two or more streams having the same nominal sample rate and assembles them into a single stream, using the timestamp and sample rate calibration information in the streams to align the inputs. Normally, each of the input streams will have had its timestamp polished by vttime.

Using vtjoin, multiple soundcards from different hosts can be combined for antenna synthesis or polarisation and bearing measurement, signals received on different sites can be merged for coherent reception or arrival time difference measurement, and streams from different signal databases can be brought together for analysis.

The streams are brought together onto one host and applied to vtjoin. For example, the following three background commands collect three streams, two are on the LAN and the third comes in from a remote site via vtvorbis

   vtcat -B ++6001 @vlf1
   vtcat -B ++6002 @vlf2
   nohup nc -ldk 6003 2>/dev/null | vtvorbis -B -d @vlf3 &
The contents of the three buffers are then combined,
   vtjoin @vlf1 @vlf2 @vlf3 - | vtfilter -e eqmap - @output
and it is convenient at this point to use vtfilter to apply an equalisation map to deal with any phase and amplitude calibration adjustments if required. Note that an output stream must be explicitly given to vtjoin otherwise it would try to use buffer @vlf3 for output.

The output buffer provides the input stream then for antenna synthesis using vtmix, arrival time difference measurement using the cross-correlation function of vtcmp, and so on.


Filtering

The general purpose filter program vtfilter is a frequency domain filter. The input signal is transformed to the frequency domain, multiplied by an array of arbitrary complex filter transform coefficients, then converted back to the time domain. Some simple filter transforms can be specified by -h options on the command line. More complex transforms can be constructed in a text file which is given to vtfilter with a -e eqmap argument. The filter transform used is the product formed from all the given -h options and the eqmap.

An automatic notch filter is built-in to vtfilter and activated with the -a option which requires a threshold argument. This is intended primarily for hum removal. An option -a th=8 should be sufficient for light hum and -a th=4 for more severe cases. Use the largest value that you can. Lower values will remove more hum harmonics but will increase the amount of reverberation. Values of -a th=3 and below will sound quite unnatural. When first started, the automatic notch will take several seconds to locate the hum harmonics and then the filter will activate. An example of the hum filter being activated can be heard here. The raw hum is present for a few seconds until vtfilter applies automatic notches. The threshold in this example is -a th=7 as the hum is fairly light and free of modulation sidebands.

You may wish to discard the initial few seconds of output when running the automatic notch to remove hum. Just follow vtfilter with a vtcat -S10 to discard the first 10 seconds of output.


Phase equalisation

To achieve antenna synthesis and coherent reception involving multiple receivers, it is necessary to compensate for differences in phase response of the receiving systems involved. The compensation is easily applied using the -e eqmap function of vtfilter. The difficult part is establishing a sufficiently good equalisation map. One useful tool for this is vtcmp with its -m pd180 and -m pd360 modes of operation. These modes compare the phase and amplitude of a two channel stream in the frequency domain and produce an ASCII output data file which reports the difference for each frequency bin.

A relatively straightforward case of phase equalisation is that of a pair of orthogonal loops. Presumably the two loops and amplifiers are identically built and so will only have small differences in their phase and amplitude responses. An end-to-end phase comparison can be performed by using a test loop placed near the antenna. The test loop carries a small signal current sufficient to produce a clear signal in the receiver outputs. Both loops should respond to this near-field signal with identical or inverted phase, depending on the location and orientation of the test loop. In this case -m pd180 will report the residual phase difference, modulo 180 degrees.

General scheme for a calibrator to align E-field and H-field phase response. The AC source is stepped up to a high voltage via the transformer, which produces an electric field. Current through the load resistor produces a magnetic field in the loop which is almost in phase with the electric field.

Off-air signals can also be used for this. Only distant sources should be used, as anything originating within 2000km or so will show considerable ellipticity of polarisation which produces large phase differences between the orthogonal loop signals. One option here may be to place the two loops in the same orientation while calibrating. Distant MSK signals for spot frequencies, and distant sferics for broad band calibration can be used. A suitable time must be chosen when there are no nearby (<2000km) sources of sferics. When using sferics for phase calibration, vtcmp must be set to do a long averaging and used in -m pd180 mode so that it doesn't matter which quadrant the sferics are arriving from.

Correcting the phase response between co-located E-field probes and loop antennas is more difficult. A rough calibration can be performed by using distant sferics and MSK stations as described above. For more precise calibration, a test source can be arranged to produce both electric and magnetic near fields in phase with one another, as illustrated on the right. The load resistor must be rated to handle all the power from the signal source and its resistance must be much higher than the reactance of the test loop so that the loop current is in phase with the E-field antenna voltage. The test source is placed near enough to the receiving antennas to obtain a good signal in both fields. vtcmp -m pd360 or vtnspec can then be used to report the received phase difference.


Arrival time differences of whistler and sferics between two sites 30km apart, produced by vtcmp -m cor and rendered with vtsidex in gnuplot pm3d

Equalising the phase of signals brought together from remote receivers is more difficult still. It requires a test source in which the test signal is synchronised to a time standard such as a PPS from a GPS. First the timestamping of the two sites must be set as carefully as possible via the c= parameter of vttime at both sites. There may be a residual systematic time difference which should be identified and corrected by a vtcat -a applied to one of the sources. The vtcmp program can again be used for these timestamp calibrations, in its -m cor mode of operation. This reports the cross-correlation between two channels. To confirm time alignment to within a fraction of a sample period it is necessary to upsample (vtresample) to a much higher sample rate before using vtcmp -m cor and fairly long averages should be taken. Once the time alignment of the two sites is reasonable, a phase comparison and correction can be made. It is probably necessary to repeat the time delay and phase correction steps iteratively, perhaps by first establishing a good time and phase alignment at a low test frequency and then working upwards through the range of interest to equalise the phase. A sanity check on the alignment between remote sites can be made by measuring arrival time differences of suitably placed distant MSK stations. vtcmp -m cor is a very good tool for ATD measurements. The image on the right shows arrival time differences of sferics and a whistler received at two sites.


Polar displays

A frame from the avi output of vtpolar -mf with 3-axis VLF reception. A sferic is arriving from the west and the zig-zag line reveals the 'polarisation error' of the bearing as a function of frequency. The central ring is some mains hum and the blob to the north-east is a beep from the Alpha navigation system.

If crossed loop receivers are available, or crossed loops and an E-field receiver, then vtpolar can be put to good use. The phase response of the receiving systems must be matched over the frequency range to be displayed and this can be achieved with the vtfilter -e eqmap function.

An example is available at this link which displays 70 seconds of activity captured with 3-axis reception. A frame from the avi is shown on the right. North is at the top with the center DC and the perimeter at 14kHz. Whistlers are coming in from slightly east of north and are seen as broad swathes due to their high elevation angle and strong elliptical polarisation. The whistlers are causing triggered emissions which are seen slightly west of north. The flashes in the north-east are beeps of the Alpha navigation system and there is sferic activity to the west and south west.

A script to retrieve raw signal data, filter and eq, and generate this AVI goes something like

   ts=start_timestamp    # Begin at this timestamp, integer seconds
   secs=length           # Length in seconds

   vtread -T$((ts-30)), /raw |
     vtfilter -a th=5 -h hp,f=400,poles=2 -e eqmap -g8 | 
       vtcat -T$ts,$((ts+secs)) |
         vtpolar -f vbr=500,abr=128,vcodec=mpeg4,format=avi -s320 \
                 -k0,14000 -mf -gv=5 -ga=5 -p96,6,E  > movie.avi
Raw data extraction is started 30 seconds early to give a bit of preamble for the hum filtering autonotch and then a vtcat crops the exact time period to display. The -p96,6,E indicates that the first channel is a loop oriented on 96/276 deg, the second channel is oriented 6/186 degrees, and the third channel is the vertical electric field.

The ffserver uplink option is currently broken.


Antenna synthesis

Signals from several receivers can be combined in order to synthesise a desired response pattern. The program vtmix performs the required mixing but first the signals must be brought together into a single stream and have their phase and amplitude responses equalised. See Multi-channel reception and Phase equalisation for details. In the following, assume this has been done and the combined signals are available in a buffer called @source. This stream might contain a pair of orthogonal loop signals and one or more E-field signals.

In these notes, the convention is used that in right-hand circular polarisation, the field vectors are rotating clockwise as seen from the source looking towards the receiver.

As a simple example, suppose @source contains two channels, one from each of two E-field receivers. Adding their signals will double the signal amplitude but only increase the system noise by sqrt(2). The command

   vtmix -c0.5,0.5 @source @output
produces the summed output with the coefficients of 0.5 restoring the original signal amplitude.

A synthesised loop signal is derived from a pair of orthogonal loop signals with a command such as

   vtmix -c0.866,-0.5 @source @output
If ch1 is an E/W loop and ch2 is a N/S loop, and the loop polarities are such that a signal from the north east is in phase in both loops then this will synthesise a loop oriented along 120/300 degrees. In general the coefficient of the E/W loop should be sin(bearing) and that of the N/S loop, cos(bearing).

The synthesised loop has a dipole response and but this can be made unidirectional if @source contains an E-field channel with its amplitude and phase correctly equalised. If loop polarities are as above with the additional requirement of being in phase with the E-field given a signal incident from the north-east, the command

   vtmix -c0.866,-0.5,1 @source @output
will produce a uni-directional response along the bearing 120 degrees. Loop signals from the back of the beam will be equal in amplitude and opposite in phase to the E-field signal and be cancelled.

A whistler train and hiss band, seen in three different polarisations. From top to bottom: RH circular, LH circular, vertical E-field.

Circular polarisations can be synthesised from the orthogonal loop signals by giving imaginary coefficients to vtmix. A coefficient of 'j' advances a signal by 90 degrees and '-j' retards a signal. Adding a delayed N/S signal to the undelayed E/W signal produces a RH circular response,

   vtmix -c1,-j @source @output
or in terms of magnitude and phase coefficients,
   vtmix -c1,1/-90 @source @output

A useful mix for looking at whistlers is to synthesise both polarisations to produce a 2 channel output, with

   vtmix -cj,1 -c1,j @source @output
Whistlers are normally either strongly RH or strongly LH polarised and therefore when the above dual circular stream is plotted on a spectrogram (vtsgram) one or other of the channels will be significantly stronger and may also have a better signal/noise ratio than the original linear polarised loop signals.

The spectrogram on the right was produced using vtmix to generate the three signals and was rendered by vtsgram. This whistler and its echo train was strongest in RH polarisation. The hiss is also RH polarised.


Startup scripts

Hosts which are running vtcard will have to run amixer or ossmix to select and un-mute the line input and set the PCM input gains. They must also ensure that the system clock is set and ntpd is running.

After vtcard is started it may take up to a minute or two before its output lock-free buffer is created. Before launching programs which use the raw buffer, startup scripts should run vtwait on the buffer in order to wait for the buffer to appear, otherwise programs attempting to use the buffer will fail immediately. For example,

   vtcard -Bvv -d hw:0,0 -b32 -r 192000 @raw,20,i4

   echo "waiting for @raw..."

   vtwait -t @raw || {
      echo "problem with buffer @raw" >&2
      exit 1
   }

   echo "@raw running"
   vtcat -B @raw ++somewhere,someport,i4
The script blocks on the vtwait until @raw is created and data appears.

It is likely that signal processing will be distributed across more than one host. Typically one or more hosts will run vtcard and possibly vttime. Another will assemble and store data with vtjoin and vtwrite and will probably host various other programs which operate on the live stream and process the output of vtread.

It is desirable that the routine processing pipelines will not collapse if one or more of the hosts involved is rebooted, and if they are all starting up, the pipelines should assemble and begin work regardless of the order in which the hosts start up. With this in mind, LAN stream connections between hosts should use the ++ syntax to specify the network connections. WAN connections would need to have loops around ssh commands in order to reestablish contact. A vtvorbis -e uplink to Icecast servers should use the -k option and vtvorbis -d programs need to be put within a retry loop.


Paul Nicholson vt11@abelian.org