Radio: RadioFax/HF Fax Reception on KiwiSDR with kiwifax.py

If you’ve seen my most recent post with the gallery of JJC Kyodo News HF faxes, you’re probably aware that I quite like receiving such signals off the air. I’ve scoured the Worldwide Marine Radiofacsimile Broadcast Schedules document from US NOAA/NWS in my quest to decode signals through the years, but propagation hasn’t always been kind and the transmissions are seldom intended to reach outside the NAVAREA the charts themselves cover.

With my recent epiphany regarding the existence of WebSDR, OpenWebRX and KiwiSDRs, I realised that my quest to receive HF faxes all around the world may have just gotten a lot easier.

What is RadioFax/WeatherFax/HF Fax?

RadioFax, also sometimes known as WeatherFax for its role in trasmitting weather analysis information to mariners or HF fax as it operates in the HF bands, is a method of image transfer over radio. It can almost be thought of as an SSB transmitter being fed with a tone at 2300Hz for white and 1500Hz for black with greys being the intermediate values. Lines can be sent at a rate of 60, 90 or 120 lines per minute or 1, 1.5 or 2 lines per second. The resulting image is often full of noise and speckles, as there is no error correction, and may be slanted as there is only limited provision for aligning synchronising the receiver clock during phasing.

I quite like receiving and decoding HF fax transmissions because they are almost like their “own” QSL card, especially if the text on the charts are readable. The weather data itself isn’t always interesting (although greyscale satellite images can be quite artistic), but the clarity of the image and the watermark text does serve as a concrete confirmation of the transmission date and station which originated the fax. It is also somewhat musical, especially if you like modem tones or repeating “screeches” at 0.5 second (120LPM) or 1 second (60LPM) intervals. It’s also a very “analog” simple mode of transmission encoding with no error correction/protection, leveraging our brain’s ability to filter noise, giving images their own “charm”.

Decoding HF Fax the Old Fashioned Way

Traditionally, radiofax can be decoded using dedicated printer receivers which you tune to the listed frequency. But these are rather expensive and clunky, intended for use onboard vessels. Instead, with the advent of sound-card based decoding software, radiofax can be decoded using software such as Fldigi (free), MultiPSK (limited resolution in free mode) or MixW (paid) amongst others. A traditional SSB-capable receiver is tuned below the listed frequency by 1.9kHz and the audio is fed into the software to decode the image. If the CPU is not being over-burdened and the sound card clock is relatively stable, a good image can be obtained after determining the slant correction.

Now that we have the possibility of remote reception through WebSDR/KiwiSDRs, the straighforward way would be to use a “virtual” audio cable to connect the output from one of these into the decoding software to produce our images. Unfortunately, things can get a little complicated.

As the internet was not really designed to ensure real-time delivery of data, the audio from online SDRs can occasionally be “choppy” due to packet loss and delays in packet delivery due to retransmissions or congestion. This results in an image which has been “chopped”, losing phase alignment almost on a continual basis, making it rather unusable and a chore to “reassemble” if you wish to try.

Something which more-commonly manifests itself with the UTwente WebSDR has been termed the “Twente Wobble” and is illustrative of another problem with audio outputs from SDRs in general. The problem is that the sample rate of the SDR itself and the sound card (real or virtual) on your computer may not be related by a “fixed” factor as they are not accurately known on starting up and can both drift relative to each other. This results in the need for irrational resampling which may result in slight shifts of the signal back and forth in time (i.e. speed up/slow down playback) to try and maintain a smooth audio flow. Unfortunately, for fax, this has a devastating effect. If you don’t do this, instead you will get gaps in your output resulting in periodic “jumps”.

This can sometimes be made worse by the configuration of the end-users’ computer – say if they have the program playing out at 48000Hz into the virtual cable and the decoder program sampling at 44100Hz, the resampling introduces an additional stage into the pipeline which degrades the signal quality and could result in underflows/overflows which cause the exact desynchronisation noted above. Worse than this, any audio processing (e.g. compression in the SDR, volume mixer scaling) will introduce further degradation to the image.

One approach that avoids most of these issues is to use the inbuilt recording feature provided by the WebSDR/OpenWebRX interface. This buffers the audio data into the browsers’ RAM and writes it out into a .WAV file. Decoding this using Fldigi’s playback feature ensures no playback sample rate mismatches, and having it recorded as just the audio data stream means there is some level of tolerance to network jitters as the recording process doesn’t have to be as close to realtime as the playback process would. It also avoids dynamic resampling as in the case of UTwente’s WebSDR. But now, reception of radiofax takes twice as long since replaying the recording for decoding is a real-time process.

The Integrated Approach

Using KiwiSDRs via the OpenWebRX interface, many of them have enabled the “fax” extension which allows you to decode fax within the web browser itself using the co-operation of the server.

In this mode, it is possible to tune a fax station and immediately begin seeing the data appear on-screen. You can also record the lines to a .png file for saving.

While this method may be sufficient for some users, for true fax aficionados, it does come with a number of significant drawbacks. The first is that it expects 120LPM faxes, thus it cannot properly display 60LPM faxes such as those from Kyodo. The second is that it doesn’t seem to have an easy slant-correction feature that is obvious. Finally, and probably my biggest objection to using this mode, is that it produces images with only 1024 pixels across when it should be 1809 pixels to ensure you get all the resolution. As a result, under good signal conditions, the faxes received through the extension can look a little blurry.

Why is kiwifax.py a Better Alternative?

The better way around all of this is similar to the above integrated approach – using partly the co-operation of the server but also working completely on the samples as data rather than playing them out on a virtual audio card and capturing them again.

The kiwifax.py program as part of the kiwiclient suite of tools definitely does this all digitally working on the demodulated audio data from a KiwiSDR directly. It takes care of connecting to a KiwiSDR and obtaining the audio data (via kiwiclient), finding the peaks and decoding the pixel values (with numpy) and writes the image data at its full resolution to a .png file (using the png library). As it is written in Python, the full source is available for modification and the program is amenable to automation, allowing for multiple parallel instances without worrying about audio routing issues as well. As it works with the raw audio data blocks, there is some tolerance to packet jitter as well. You could even run this on a Raspberry Pi remotely over SSH in a screen session to keep it running even after logout, for example.

The downside to kiwifax.py compared to the other approaches is a lack of graphical interface. I personally find this to be a feature, but it also makes it difficult when trying to tune the slant correction factors or trying to see what is being received. Still, on the whole, the benefits are worth the slight user unfriendliness.

Getting Started

In order to get kiwifax.py working, you need a working Python 2 installation with numpy. If you are on Windows and you haven’t already got an existing Python 2 installation, I would recommend you install WinPython 2.7.10.3 as this contains a self-contained installation of Python and numpy libraries. Once you install it into a folder, open WinPython Command Prompt.exe to access the command prompt.

If you are on Linux, you can check if you already have Python installed by opening a terminal and executing the command “python” –

Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

If you receive something like the above, then congratulations, you already have Python 2 installed. Now you need to check if you have numpy installed, so execute “import numpy as py” and if you are returned to the >>> prompt, then you already have it installed.

>>> import numpy as py
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy

If you don’t have numpy, you can attempt to install it with “sudo apt-get install python-numpy”. To exit the Python shell, use either exit() or Crtl+D.

Now you have a working installation of Python 2 and numpy, you will need to obtain the kiwiclient package. The most up-to-date at this time seems to be the branch maintained by jks-prv here. You can either visit the site, download ZIP and unzip it to a folder, or if you have git installed, use “git clone https://github.com/jks-prv/kiwiclient” to download a copy of the code.

Open up the WinPython Command Prompt or the Linux terminal and change directories to where the kiwiclient code lives. You should be able to issue the command “python kiwifax.py -h” and obtain the help listing below:

Usage: kiwifax.py [options]

Options:
  -h, --help            show this help message and exit
  -k SOCKET_TIMEOUT, --socket-timeout=SOCKET_TIMEOUT, --socket_timeout=SOCKET_TIMEOUT
                        Timeout(sec) for sockets
  -s SERVER_HOST, --server-host=SERVER_HOST, --server_host=SERVER_HOST
                        server host
  -p SERVER_PORT, --server-port=SERVER_PORT, --server_port=SERVER_PORT
                        server port (default 8073)
  --pw=PASSWORD, --password=PASSWORD
                        Kiwi login password (if required)
  -q, --iq              IQ data mode
  -f FREQUENCY, --freq=FREQUENCY
                        Frequency to tune to, in kHz (will be tuned down by
                        1.9kHz)
  --station=STATION, --station=STATION
                        Station ID to be appended to file names
  -F, --force-start     Force the decoding without waiting for start tone or
                        phasing
  --force-offset=FORCE_OFFSET, --force_offset=FORCE_OFFSET
                        When force decoding, apply this tuning offset (bins).
  -i IOC, --ioc=IOC     Index of cooperation; default: 576.
  -l LPM, --lpm=LPM     Lines per minute; default: 120.
  --sr-coeff=SR_COEFF, --sr_coeff=SR_COEFF
                        Sample frequency correction, ppm; positive if the
                        lines are too short; negative otherwise
  --max-height=MAX_HEIGHT, --max_height=MAX_HEIGHT
                        Maximum page height; default: 2300.
  --dump-spectra, --dump-spectra
                        Dump block spectra to a CSV file
  --dump-pixels, --dump-pixels
                        Dump row pixels to a CSV file
  --dump-histo, --dump_histo
                        Dump pixel intensity histograms to a CSV file
  --iq-stream, --iq_stream
                        EXPERIMENTAL: use IQ stream instead of audio
  --tlimit=TLIMIT, --time-limit=TLIMIT
                        Record time limit in seconds

If so, congratulations, you’re basically ready to start using kiwifax.py. If not and instead you receive:

  File "kiwifax.py", line 767
    print "Failed to connect, sleeping and reconnecting"
                                                       ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Failed to connect, sleeping and reconnecting")?

This is because you have Python 3 and not Python 2 installed. Unfortunately, the code has been written targeting Python 2 and requires a number of changes to make it Python 3 compatible.

Using & Modifying kiwifax.py

To use kiwifax.py, you can refer to the help as noted above, but basically you will need at minimum a frequency to tune to and a KiwiSDR server to use as the source. Assuming a frequency of 10000kHz and a server of abc.com, you would invoke it with “python kiwifax.py -f 10000 -s abc.com”. Setting the other options may be important in case you have a server on a non-default port, a password on the server, a different line rate, need to override the maximum height or wish for the data to be exported in a different format.

When it is running correctly, it should print its debug messages to the console including the status (idle, printing, phasing, starting, stopping), detected frequency peaks, signal strength and an ascending block number with every block of data received from the KiwiSDR server.

In this situation, the script is working and faxes will be received and written into the folder where the script is with a default name of YYMMDDTHHMMZ_F.png (e.g.20190109T0028Z_7396900.png). It works by sensing the Automatic Picture Transmission (APT) start and stop tones to decide when to start the fax and when to end the fax, terminating also if it reaches the maximum lines (default 2300, but can be set using –max-height= parameter).

There are a few annoyances to kiwifax.py, but because it is open source, anyone can modify it to suit their needs. The first is that it generates a log file that has all of the console output – this causes needless writes to flash memory and can wear out microSD cards (e.g. running on a Raspberry Pi) or SSDs.

As a result, it’s possible to just comment out the file-handler lines so that logging to file is essentially disabled.

Another annoyance is that the default IDs and location settings are probably not what you would wish to send out to the KiwiSDR you are connecting to – so you should probably change these to set the ID to something more descriptive – the last two lines of this function.

Of course, inevitably, you are going to receive some faxes with slants and this may differ depending on the server you have selected as some may not have GPS reference and will be running off a “free running” clock. As a result, it may be very useful to modify this array to add your own corrections, iteratively changing the value until the picture looks straight (enough).

In this case, I added a line to use the VN/SWL KiwiSDR in Hanoi, Vietnam with a correction for HSW64 – Thailand’s HF fax station. Note that other receivers if GPS-synchronised will be able to use the same slope – which is one big advantage of having the GPS in the first place.

This was the HSW64 test-chart prior to slant correction. After adding it in, receiving it a few days later using the “-F” option (as it missed the start tone), it was much straighter although missing part of the chart and phasing.

For receiving Kyodo which has mixed 60LPM and 120LPM broadcasts, the script cannot detect the line rate, so it’s best to set it with -l 60 and have 120LPM faxes received “doubled-up” side by side as the majority of the faxes are 60LPM. By default with zero correction going through BrisSDR, the skew goes the other way but only very slightly.

But the result is extremely sharp and requires no complicated virtual-audio-cable configuration, sample rate matching and suffers much less from image “chop” due to delayed packets. However, I have still experienced chop with kiwifax.py, possibly due to long network latencies, CPU exhaustion on my Asus Tinkerboard appending lines to long faxes or high packet loss scenarios.

Another frustration you may find is that the APT does not properly trigger on the start/stop tones as different stations sometimes use non-standard tone combinations. As a result, it might be useful to see if your target station has any technical specs and modify the code accordingly to better-match what is transmitted over the air. Otherwise, you might find faxes may start receiving correctly but only stop once reaching the maximum height. If another fax were to start, then you would probably find the reception terminates in the middle of the second fax and without another start-tone trigger, the remainder of that fax would be lost. The -F force option starts decoding immediately until a stop tone is encountered or until the maximum height is reached, but does not run a continuous rolling decode. That being said, this might be something you can easily modify the code to do.

While it is tempting to monitor frequencies for the existence of HF fax signals around the clock, please be mindful that public KiwiSDRs are for the benefit of the entire SWL/ham community and should be shared rather than monopolised. It’s best to do your research and limit your use of the receiver to either your own KiwiSDRs, those you have permission to use or use it only for limited periods with public KiwiSDRs. Unfortunately, I don’t think there is an analog of this for the WebSDR … so I am still using the record-and-decode .WAV method with that for now.

Conclusion

HF fax is a relatively old, slow and simple mode of sending an image over the radio. Despite the simplicity of the mode, it is still used to deliver weather and news to mariners who may have no other method of communication. The mode has its charms in noisy, slanted but readable images and is one of my favourite types of services to receive on the air as almost everything is dated and identified with the transmitting station’s logos/name.

Traditional decoding methods were very much time-sensitive real-time processes, meaning that any irregularity in packet jitter that overwhelms the buffer will cause desynchronisation of the receiver and a “chopped” image. This can be made worse by sample-rate mismatches which may result in wavy images on dynamic resampling or further “chopping” as the buffer underruns or overruns.

A much better way is to deal with this purely in the “data” domain, which kiwifax.py does. The Python 2 code is fully provided for modification, highly suited for automation and can even run on a Raspberry Pi. By using it, we can obtain sharper images with less hassles and less potential for image “chopping”. Unfortunately, slant tuning becomes slightly more difficult lacking a GUI or preview and there are a few niggles with debug output and client identification by default that can easily be fixed. It is a very useful tool for the HF fax enthusiast, which offers better decode resolution than the OpenWebRX extension.

The kiwifax.py script is only part of the whole kiwiclient package which also includes kiwirecorder.py which allows for automating recording from a KiwiSDR. In fact, KiwiSDRs and the OpenWebRX ecosystem have even more nifty features to offer, some of which I’m only just discovering. When I have the time, I might write a few articles covering some of those other cool features (such as TDoA).

About lui_gough

I'm a bit of a nut for electronics, computing, photography, radio, satellite and other technical hobbies. Click for more about me!
This entry was posted in Computing, Radio and tagged , , , , . Bookmark the permalink.

2 Responses to Radio: RadioFax/HF Fax Reception on KiwiSDR with kiwifax.py

  1. Benson says:

    Excellent article about HF fax reception in general and using kiwifax. Information on setting up Python to be able to modify kiwifax is very helpful.

    A really useful further addition would be a time scheduler (in python) to limit the access period and not to tie up public Kiwi receivers where the user specifies the start & stop time and Kiwi IP addresses for the attempted download of certain charts.

    Thanks, Ben

    • lui_gough says:

      Dear Ben,

      Indeed, that would be a nice feature to have – although I’m not really experienced with Python myself, so I probably wouldn’t be making the change myself.

      Instead, you can schedule the start using something like Task Scheduler under Windows and cron under Linux, but the downside is that if the server is down then the code exits with exception and does not re-attempt connection (and if busy, it will start printing late when you manage to grab a slot). As to actually making a graceful finish, I would think that modifying the maximum lines handler to not start a new roll/change state but instead exit() might well allow kiwifax.py to exit after printing a given number of lines – e.g. 30 minute fax * 120 lines/min = 3600 lines. If the signal is good, regular kiwifax.py could be used and possibly an external script can be used to kill the specific process by calling with cron when it is known that the chart transmission has finished. Killing while still printing has occasionally resulted in corrupted PNG files, thus is not recommended.

      I agree that it would be nice not to tie-up slots (something I’ve been guilty of doing to create the recent series of Radiofax postings) … but sometimes there are advantages and assurances that come with continuous monitoring that can make it worthwhile (especially in discovering unlisted test-charts or for stations with known schedule errors/no known schedule).

      – Gough

Error: Comment is Missing!