Having recently reported receiving QSL cards for the Ghosts in the Air Glow program, I thought it would be nice to report on a little project I undertook just-in-time for the last transmission that occurred in October 2022. While ultimately not successful in receiving the program, the project is definitely useful in other ways.
The Problem
With the urban city environments suffering ever-increasing noise floors in the HF/shortwave bands, I’ve all but given up trying to receive distant and far-away signals from home. Instead, I’ve been spoiled by being able to take virtual DXpeditions through the plethora of online SDRs that have been set-up by generous operators with access to excellent radio environments.
For KiwiSDRs, I’ve enjoyed the use of kiwirecorder.py and kiwifax.py which have fulfilled a need for tools that allow for automated monitoring. While such clients will not work with all KiwiSDRs and lately, blocklists have been implemented to kerb abuse, these tools have done a fantastic job in allowing me to receive Ghosts in the Air Glow this time around, and underpin the numerous radio-fax galleries on this site.
There is, however, one target from which I would like to attempt reception of Ghosts in the Air Glow from – the venerable WebSDR at University of Twente. This has been an excellent receiver, one capable of serving hundreds of users simultaneously with relatively good reception.
Unfortunately, a quick search of the internet was not able to uncover any tool that could make automated recordings from the site. Instead, it seems that I would have to leave the browser window open to keep a recording running, wasting bandwidth and risking running out of RAM as well.
This was not an ideal situation to be in … but it seems others may have their own solution –
Looking at the list of clients, I saw a mysterious user named “get” with many many instances, seemingly spaced at near-regular intervals and persisting for days on end. It seems they are doing some monitoring as well … This inspired me to look for my own solution.
My Quick and Dirty Solution
My first thought was to take apart the code for the site and behave just like a user. After all, this seemed to be the best way to do this … but I came across this in the header of one of the Javascript files:
//WebSDR HTML5 client side - Copyright 2013-2020, [email protected] - all rights reserved //Since the intended use of this code involves sending a copy to the client computer, I (PA3FWM) hereby allow making it available unmodified, via my original WebSDR server software, to original WebSDR clients. Other use, including distribution in part or entirety or as part of other software, or reverse engineering, is not allowed without my explicit prior permission.
While the Javascript was nice and readable … I’m not allowed to reverse engineer it. I can understand the desire to avoid this, so I won’t touch the Javascript nor emulate it to respect PA3FWM’s wishes.
Instead, I’ll look at the HTML instead, which doesn’t contain such a disclaimer and behave more like a real user. My solution was simple – use a web-browser and simply automate it. Having just a weekend to write it up, I did some research online and settled on using Python 3, Selenium WebDriver, Mozilla Firefox and the matching Geckodriver as my foundation.
To make things work, I would make Javascript calls equivalent to a user pushing buttons, or navigate to text fields, enter in keys and/or click buttons. I had no consistency in my approach as some things seem simpler via one method versus the other and speed was of the essence.
The Result
In the end, this resulted in a script I call websdr-recorder-v04-uni.py which can be downloaded here, with the full listing in the appendix. It is not complete – for one, there is a deprecated call that should be fixed, it doesn’t gracefully save recordings on premature termination and there are hardcoded wait times in the script. It also assumes you’re using the UTwente WebSDR and none other.
But it does work and it has a simple interface that is somewhat similar to kiwirecorder.py. In the above example, I am running this under WinPython.
By default, it runs headless, so you won’t see it (or hear it) doing its work. But it gets the job done as long as you let it run to completion. It generates a random username of the format “websdrrec_XXXXXX” just to be nice and allows for precise configuration of filters as well. One funny thing is, while I do “begin audio”, Firefox will mute it as it saw no real user interaction, which is why the program works quietly.
But it doesn’t have to run headless either – you can watch it do its thing.
It runs on Linux too – I’ve used it on Ubuntu Server (on my VPSes), Ubuntu Desktop and even under Windows Subsystem for Linux.
But because WSL as an environment is just like a VM with Linux, you’ll have to install the Linux version of Firefox and Geckodriver … but other than that, it should work.
Using it for Yourself
I was debating with myself after developing this tool whether to release it openly, given the potential for abuse. But I decided that it was perhaps trivial enough that it would be best just to “give it away” under CC BY 4.0 and let the internet do its thing. After all, there are much easier and worse ways to cause abuse – this thing chews RAM so spawning a large number of copies on a VPS is nigh impossible so wouldn’t be useful for attacks. But what it is useful for is saving resources by not having users leave windows open when they don’t need it just to catch a program while they’re in bed.
So, how does one get started with the tool?
- Download the code and extract it somewhere.
- Install Python 3 – under Windows, you could use WinPython, or use Windows Subsystem for Linux and treat it like Linux. Under Linux, this is as simple as going to the prompt and issuing sudo apt-get install python3 python3-pip.
- Get Selenium – open up a terminal (or WinPython Command Prompt) and issue pip3 install selenium or pip install selenium depending on your platform.
- Install Firefox – under Windows, download it from here. Under Linux, issue sudo apt-get install firefox
- Download and extract the latest Geckodriver version for your platform from here – for Windows, you’d want a file with “win32” in it. For Linux, choose a file with “linux64” if on a x86-64 64-bit computer, or “linux32” if on a x86 32-bit computer, or “aarch64” if on an ARM-based 64-bit computer. To simplify things, you can extract it to the same folder as the code.
- Run the code and do a test recording – it may be most instructive to run it in non-headless mode to see it do its thing. You can try python3 websdr-recorder-v04-uni.py -t 60 -z 0 or python websdr-recorder-v04-uni.py -t 60 -z 0 (depending on your platform) to create a 60-second recording with default username, destination, frequency, mode and passband. If successful, you should see the browser pop-up and do its thing – there are a few hard-coded waits, but once it completes, you should find a recording .wav file in the directory where you invoked the script from.
In case of problems, check the paths to Firefox and Geckodriver are correct and check the execution permissions (try a chmod +x geckodriver).
For unattended recordings, it can be possible to schedule a cron job with a command similar to cd /some/directory && python3 websdr-recorder-v04-uni.py -f 1234 -m usb
Conclusion
Not being able to record in an automated fashion has been a frustration of using WebSDR for a while. Inspired by the evidence that others have somehow automated the use of WebSDR and trying my best to keep in line with the “no reverse engineering” condition listed in the JavaScript files only, I embarked on a simple weekend project to allow me to try catching the Ghosts in the Air Glow program without leaving WebSDR windows open for prolonged periods, risking running out of RAM, wasting precious LTE bandwidth and electricity.
In the process, having come from a background of nearly no JavaScript experience, minimal HTML experience and no experience in automating web browsers, I started to become comfortable with using Selenium WebDriver to automate Firefox through Geckodriver which is a very powerful tool to have.
While I met my goal of trying to receive the program, unfortunately, the reception conditions were poor and no usable signal was received in the end. The recorder itself also has a few things which could be improved – by my goal was simply to get something working, not something perfect. To that end, I’m happy to report that I succeeded.
Appendix: Code Listing
The code for websdr-recorder-v04-uni.py is listed below. Alternatively, you can download it in a ZIP file.
from selenium import webdriver
from optparse import OptionParser
import os
import time
import random
import string
#TODO: Fix Aborting Recording Nicely
#TODO: Fix Ugly Hardcoded Times for Page Load/Recording Save
print("WebSDR Recorder by Gough Lui (goughlui.com) v0.4")
print("October 2022 - Use at your own risk! Licensed under CC BY 4.0")
print("Requires Selenium, Firefox, Gecko WebDriver, Python 3")
print("")
parser = OptionParser()
if os.name == "nt" :
print("You are running a Windows-based platform ...")
parser.add_option("-b", "--binary-location", dest="bloc", default=r"C:\Program Files\Mozilla Firefox\firefox.exe",
help="Firefox Binary Location")
parser.add_option("-w", "--webdriver-location", dest="wloc", default=r"geckodriver.exe",
help="Geckodriver Binary Location")
else :
print("You are running a POSIX-based platform ...")
parser.add_option("-b", "--binary-location", dest="bloc", default=r"firefox",
help="Firefox Binary Location")
parser.add_option("-w", "--webdriver-location", dest="wloc", default=r"geckodriver",
help="Geckodriver Binary Location")
parser.add_option("-n", "--username", dest="uname", default="websdrrec_"+"".join(random.choice(string.ascii_uppercase + string.digits) for _ in range(6)),
help="Username (Default websdrrec_XXXXXX)")
parser.add_option("-f", "--frequency", dest="f", default="10000",
help="Tuning Frequency (kHz)")
parser.add_option("-m", "--mode", dest="m", default="usb",
help="Tuning Mode (cw, lsb, usb, am, fm, amsync) (Default: usb)")
parser.add_option("-t", "--recordtime", dest="t", default="3660",
help="Record Length (s) (Default: 3660s)")
parser.add_option("-u", "--upper", dest="hf", default="0",
help="High-Frequency Cut-Off (kHz) (optional)")
parser.add_option("-l", "--lower", dest="lf", default="0",
help="Low-Frequency Cut-Off (kHz) (optional)")
parser.add_option("-o", "--output-directory", dest="dld", default=os.getcwd(),
help="File Download Directory (Default: Current Working Directory)")
parser.add_option("-z", "--headless", dest="z", default=1,
help="Headless Mode (Default: 1)")
(options, args) = parser.parse_args()
print("Launching Browser ...")
opts = webdriver.firefox.options.Options()
opts.binary_location = options.bloc
if options.z == 1 :
opts.add_argument("--headless")
opts.set_preference("browser.download.folderList", 2)
opts.set_preference("browser.download.manager.showWhenStarting", False)
opts.set_preference("browser.download.dir", options.dld)
opts.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream")
browser = webdriver.Firefox(options=opts,executable_path=options.wloc)
browser.get("http://websdr.ewi.utwente.nl:8901/") # Hard Coded URL
time.sleep(10) # Hard Coded Delay for Page to Load
browser.execute_script("soundapplet.audioresume();") # Start Audio, but often muted as no user
time.sleep(0.1)
browser.execute_script("setview(3);")
unfield = browser.find_element(webdriver.common.by.By.NAME, "username")
unfield.send_keys(str(options.uname)) # Set username
unfield.send_keys(webdriver.common.keys.Keys.RETURN)
time.sleep(1)
frfield = browser.find_element(webdriver.common.by.By.NAME, "frequency")
frfield.send_keys(webdriver.common.keys.Keys.CONTROL+"a")
frfield.send_keys(str(options.f)) # Set frequency
frfield.send_keys(webdriver.common.keys.Keys.RETURN)
time.sleep(1)
browser.execute_script("set_mode('"+str(options.m)+"');") # Set Mode
time.sleep(1)
if not (options.hf == "0" or options.lf == "0") :
browser.execute_script("window.lo="+str(options.lf)+";")
browser.execute_script("window.hi="+str(options.hf)+";")
browser.execute_script("updbw();") # Set Filter Parameters
time.sleep(1)
browser.execute_script("record_click();") # Start Record
recbegin = time.time()
if not (options.hf == "0" or options.lf == "0") :
print("Recording started ... "+str(options.f)+" kHz "+str(options.m)+" mode "+str(options.lf)+"-"+str(options.hf)+" kHz passband by "+str(options.uname)+" for "+str(options.t)+"s.")
else :
print("Recording started ... "+str(options.f)+" kHz "+str(options.m)+" mode default passband by "+str(options.uname)+" for "+str(options.t)+"s.")
while (time.time() < recbegin + float(options.t)) :
time.sleep(1)
browser.execute_script("record_click();") # End Record
print("Recording stopped ...")
svfield = browser.find_element(webdriver.common.by.By.LINK_TEXT, 'save')
svfield.click() # Click Save Button
print("Saving recording ... ")
time.sleep(60) # Fixed Delay for Save to Complete
print("Closing down ...")
browser.close()
browser.quit()
print("Job Done!")
