Over the past few weeks, we had experienced a few power outages and it came to my attention that the CFL in my room might be about to go. It had a characteristic “brightness” variation which seemed to pulsate from time to time, which is common for fluorescent tubes reaching their end of life.
But the problem was, this happened occasionally for nearly a month and the CFL still didn’t die. Increasingly, I became aware that my computer’s power supply “hummed” along with the flicker … as if in time.
It would come and go in “trains”. It sounds like this … Maybe you’ve heard it before?
Looking for the Culprit
Having made the connection between the sound from the power supply and the flickering of my CFL, it was apparent that this was more of a power quality issue. Maybe there was a burst of harmonic noise or some cyclic instability in the supply voltage.
I decided to grab a junk-box 9v AC 1000mA adapter I got from a ham-fair ages ago, and hook my PoScope2 Basic to it and see what was up …
At first, all I saw was a stable 50-hz-ish sinusoid-ish waveform. The flattened tops are likely not to be as bad in reality – it’s probably due to magnetic hysteresis or core saturation in the transformer.
I waited … and I was rewarded …
Wow. The waveform grew fur! We saw about a 0.6v pk signal riding on the roughly 16.5v pk mains waveform. This would put my estimate of the ripple voltage at about 8.7v RMS although it’s probably not accurate. Instead, lets take a look in the frequency domain.
Normally, the mains looks like this – the harmonics are probably not as bad as it makes it seem – mainly due to transformer distortion probably.
The signal, when it is on, presents itself as a peak at about 1050Hz (a little left of it actually). The amplitude relative to the 50Hz signal is about -24dB which would put the ripple signal at … 15.14v RMS. Hmm. A bit of a difference depending on how one interprets the data (as expected given the crudeness of the equipment) but that’s a sizable amount of ripple.
The Culprit is Identified!
It turns out that this was enough information to identify what the culprit was – Zellweger Off-Peak Signalling. In fact, the Wikipedia article linked even identifies parts of Sydney as having 1042Hz ripple signalling. I’m positive that this is the cause of the flicker.
This type of signalling superimposes coded ripple signals onto the mains by injecting it into the HV network and letting it propagate down to the LV network. It is used for load control, to switch off off-peak storage water heaters during peak times and thus allow for better utilization of the generation and network facilities. It’s also used as a last-line-of-defence load-shed in case of network emergencies.
Here’s a few videos RODALCO2007 shot of various Zellweger ripple plants. I found them quite interesting to see just what was producing such signals.
As it turns out, further research shows that people are having some annoyance with this signal. Ceiling fan noise, dimmer light flickering and solar inverter drop-outs are just some of the problems the signalling seems to cause, especially when the signal is too strong.
From some digging, it seems AS61000.2.2:2003 R2013 (Electromagnetic Compatibility (EMC) Part 2.2: Environment – Compatibility levels for low-frequency conducted disturbances and signalling in public low-voltage power supply systems) covers ripple signalling levels. Section 4.2 covers voltage fluctuations and flickers and section 4.10.2 covers Ripple Control Signals which suggests that the injected signal “should be in the range of 2% to 5% of nominal supply voltage,” “but resonance can cause levels to rise to 9%”. I suppose what I’m seeing is pretty much within spec when the last section is taken into account.
The effect of resonance on the signal levels can cause signals to be stronger than at the point of injection – a point well demonstrated by this paper by Bhattacharyya, S., Cobben S. and Toonen J. titled Impacts of Ripple Control Signals at Low Voltage Customer’s Installations as well as this other paper by Elphick, S., Perera, S., and Browne, N. titled Ripple Signal Amplification: Measurement, Modelling and Mitigation. Interestingly, this last paper was done with Integral Energy’s Power Quality & Reliability Centre, now Endeavour Energy (my local utility provider).
Unfortunately, in the case of Endeavour Energy, there doesn’t seem to be any technical information available about their signalling itself, nor the expected levels. I suppose I will just have to live with it … or maybe try and find out more about it by myself!
Analyzing the Mains
WARNING: Mains electricity can be lethal if handled improperly. Further to this, mains electricity can contain transients which can cause arcs and severe damage. Be careful, and take responsibility for what you do, because I will not be held responsible for anything you may or may not achieve!
Analyzing the mains properly and on a budget isn’t easy. The best solution I came up with is to use an AC transformer (I had a spare 9v AC 1A transformer) and voltage-divide the output for feeding into a sound-card for continuous monitoring. Unfortunately while the PoScope2 basic can do “one-off” analysis, it’s not suitable for continuous monitoring, hence the need to use a sound card instead.
The interface I built looks like this. It’s a bit “back of the envelope” – the resulting voltage is probably higher than expected since the unloaded transformer puts out a higher voltage than I guessed … It’s mostly a choice of convenient components I had lying around on my desk. Key features are low power dissipation, isolation from the mains, short-circuit-safe on the output. The exact output level isn’t critical, as long as it’s below the line-level input threshold (otherwise, you may find it clips when acquiring from the sound card, or causes damage to it). Some sound cards are more tolerant than others.
One has to be aware of measurement problems:
- It’s not possible to accurately measure harmonic component amplitudes as the transformer will distort the waveform due to magnetic hysteresis. This in itself will amplify certain harmonics.
- Further to the above, the transformer itself may have a frequency response limitation as well which may lead to underestimating higher harmonics.
- Measurement of the mains frequency will depend on the stability of the clock-source used in your sound card. In many cases, it’s not *that* stable (drifting with temperature, mainly), so don’t necessarily take your measurements for granted.
- Measurement of the fundamental amplitude (voltage) relative to harmonic components will also be affected by the frequency response of your sound-card, some of which begin to roll off around the 50Hz area.
- Measurement of voltage amplitude can be possible, if calibrated beforehand at a fixed input volume level on the mixer, but the resolution (at most) is limited by the sound card.
To build one, it’s super easy. I just grabbed spare resistors, transformer and cut an RCA lead into two – it’s much less useful given the new “digital” age we’re in. Using that, and a USB sound-card with RCA line-in and you’re set. For added safety, this is plugged in after a surge protector which should hopefully catch any significant transients.
It can be built very neatly with some forethought …
… the finished product is something which feels odd to electronics guys – a transformer ending in an RCA plug!?
Viewing the Mains Through the Interface
The interface is nothing “high-tech”, but the information gathered seems to be quite surprising. For signal plotting, I used Spectrumlab with tweaks to the settings. For example, I’m living adjacent to many industrial consumers of electricity – and some of the “dirtyness” can be seen once we exaggerate the amplitude scale somewhat.
All this noise, and we don’t even normally see it. Now it’s a matter of waiting for the signal …
The Signalling Itself – K22/Decabit Hybrid
We managed to capture one train of the signals, which reveals an interesting structure.
The signal is at 1042Hz, just below the 1050Hz (21st harmonic) line. You can see a start-tone, pause and then on-off-keyed pulses indicating messages. There are long on-periods occasionally.
Upon further investigation, the encoding scheme was determined to be K22/Decabit Hybrid system. There really isn’t much information about this scheme online – although the premier resource here gives a great outline of how the signal is interpreted, and this other resource provides the bit-pattern mapping.
In my experience, there is a subtle error in the timing diagram provided. The signal consists of a ~ 5s start tone, ~7.5s off synch slot, followed by Channel 1 to 22 at 7.5s each slot. Not Channel 0.
The signal itself lasts for a nominal 177.5s, making it a very slow form of communication.
Each slot itself starts with a 900ms quiet time, 600ms start bit, followed by ten 600ms symbols making up the 7.5s slot. This is clearly why it is called decabit – it consists of ten bit messages. The ten bit messages encode about 8-bits of data, following the rule of 5 on and 5 off. Any message which violates this is invalid, but may be recovered from a single bit error.
Further to this, the signal reference from Orion (a NZ utility) provides a pretty good overview of Ripple Signalling and how it’s used. If only our utility were to publish something similar.
However, a closer look shows our signals aren’t only decabit. For example, this capture …
Notice how only the first slot contains a decabit message. The rest of the slots either contain all 7.5s tone or all 7.5s silence. The lack of a defined silent period between slots gives away the fact it’s not decabit coded.
I believe this is the older K22 scheme, where a presence of tone in the slot causes the devices to be turned on, and the transmission of the 3-minute message without tone in that slot causes it to be turned off.
It’s not easy to decode the messages just by looking at a spectrogram – and while chucking the audio (warning RAW audio, high 50Hz component – speaker damage possible!) into an audio editor and doing a high-pass filter at 250Hz produces an envelope representing the signal, it’s still not easy to read.
Notice how the amplitude changes slightly from slot to slot – this might have to do with loading differences on the network from appliances being switched on and off.
To better analyze the system, I will have to develop a decoder – but with very little DSP coding experience, it’s not going to be too easy.
MATLAB: The Saviour
I’m not a DSP nut, but I have a little experience with MATLAB. It’s quite nice, because MATLAB help is mostly helpful – as long as you have the algorithm sorted, you can get it to do almost everything for you.
It took me several stages to get a “useful” Zellweger K22/Decabit decoder, although it’s still very limited.
- The first task was to develop a system in MATLAB which could read samples from a sound card on a nearly continuous basis, and determine the presence/absence or magnitude of a 1042Hz signal in the data.
- The second task, was to alter this system to be able to recognize the tone at given intervals corresponding to expected bit times and output a RAW binary train representing the presence or absence of tone at the given times. This will be logged to file along with reception time.
- The third task was to try and get this binary train separated into decabit commands and K22 channel states.
The methodology for development sounds simple but the implementation and deviations from expectation proved to be quite problematic. Some challenges included a non-standard “sync” or “silence” slot timing on the transmissions made by my utility.
The code I ended up with works by doing the following:
- Record 100ms block from the sound card, check if 1042Hz signal is above given threshold. If not, repeat until it is.
- Note the system time and the presence of the signal by fft, and begin recording a 177.5 second audio block for analysis.
- Analyze audio in 100ms chunks. Start by sliding the 100ms window across until the “start tone” falls below a given threshold to achieve “time alignment”.
- Skip the “rest” time (nominally 7.5s, although it was measured to be less on my utility’s injector, so the correct value for my utility is used in the code).
- Begin iterating each channel “block” of 7.5s. Skip 900ms quiet time, look at each block of 6 x 100ms chunks. If 1042Hz tone is present above fixed threshold in 5 or 6 of the chunks, declare binary 1, else 0. Run for a total of eleven bits per channel (sync + 10 data bits).
- Open output file and dump K22 states. For all channel slots where filled by 11 binary 1’s or 0’s, declare K22 state ‘1’, or ‘0’ respectively. For all channels otherwise filled, denote with ‘D’ for suspected decabit slot.
- Interpret decabit states from data train for each slot with sync bit 1, and not all bits 1, find match from array of valid decabit-on and decabit-off signal definitions. Print command value and ON/OFF state, else print invalid for no match.
- Print the raw binary train (for debugging purposes – often we have extra or missing bit – no attempt is made to make one-bit corrections), close file, clear some variables and restart from beginning.
It is clear that this algorithm is not optimized for CPU usage (e.g. fft instead of goertzel), nor is it optimised for decoding robustness (e.g. decabit suspected by non-all-ones or non-all-zeroes case, but no sync bit, or missing 1/0). It’s very much “fixed time” with sound card providing time reference, and is susceptible to injector signal timing variance, although there is *some* leeway for optimizing sensitivity (majority-detection provides better likelihood of getting correct-result despite slight misalignment).
As I’m doing this for my own interest, in an ad-hoc fashion, I’ve written my m-script comment-less and messily. At times, I’ve used a counter starting from 0, and others, a counter starting from 1 (as I’m more familiar with 0-indexed arrays rather than 1-indexed arrays that MATLAB uses). Very specific magic numbers are used for timing, and these are often number of samples where the sound card produces the timing reference.
I think it’s clear that I’m no MATLAB expert, and it may be much easier to write an even better decoder. But it was a learning experience for me, to get used to some of the syntax and get something working without needing to think too hard about how to get it implemented.
Unfortunately, after running the code for a day or so, the memory usage grows as if there’s a memory leak and the CPU utilization jumps quite dramatically – I think it maybe an issue with MATLAB as I can’t see any “continual” declaration of new variables. I typically re-use old variables, so I don’t think it should grow in memory usage but it does …
What Did I Find?
A few interesting tid-bits. It seems that transmissions are scheduled in slots starting at xx:25:15 and xx:55:15, although not at all times. There are unscheduled transmissions as well, which occur at different times each day, likely related to street-lighting by my observation and may be fed by a light-sensor at the substation for pilot-wire operation or “global” street-light enable/disable.
Note that the scheduled slots are not always transmitted but seem to be transmitted a majority of the time in the few days I’ve been monitoring. The slot at 16:55:15 is always a decabit heavy message which turns OFF almost all known channels in preparation for the evening peak:
2014 01 30 16 55 15 : K22: DDD0D0D0DDD00000DD000D DECABIT: CH1:25-OFF CH2:24-OFF CH3:18-OFF CH5:12-OFF CH7:11-OFF CH9:10-OFF CH10:8-OFF CH11:6-OFF CH17:4-OFF CH18:3-OFF CH22:2-OFF RAW: 10100010111101100110011011001101000000000000101011001010000000000010101010101000000000001010100011110011100101100110011010000000000000000000000000000000000000000000000000000000111110001001011010010100000000000000000000000000000000010110010101 2014 01 31 16 55 15 : K22: DDD0D0D0DDD00000DD000D DECABIT: CH1:25-OFF CH2:24-OFF CH3:18-OFF CH5:12-OFF CH7:11-OFF CH10:8-OFF CH11:6-OFF CH17:4-OFF CH18:3-OFF CH22:INVALID RAW: 10100010111101100110011011001101000000000000101011001010000000000010101010101000000000000010000001110011100101100110011010000000000000000000000000000000000000000000000000000000111110001001011010010100000000000000000000000000000000010110010001
I’ve separated it into separate lines – normally it’s logged together, but you will note a few issues with the decode in the second time where there’s a sync bit missing and another bit missing in Channel 22.
The actual channel layout is as follows:
K22-Channel Layout: DDDKDKDKDDDKKKKKDDKKKD 11 Decabit Channels + 11 K22 "Legacy" Channels
The K22 channels in use are 4, 6, 8, 12, 13, 14, 15, 16, 19, 20, 21 and the observed decabit commands are so far: 2, 3, 4, 6, 8, 10, 11, 12, 18, 24, 25, 50, 100, 103, 110.
A careful look appears to show that K22-Channel 8 and Command 50 are street-lighting related – just today I saw it flick off at 6:17:16 and then back on at 6:31:18 before back off at 6:35:29. The channels are disabled by 6:55:15 normally. Some commands are sent “redundantly” in the sense an “off” channel is commanded “off” again and vice versa.
2014 02 02 06 17 16 : K22: D001010000011111001110 DECABIT: CH1:50-OFF RAW: 10000111101000000000000000000000011111111111000000000001111111111100000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111000000000000000000000011111111111111111111111111111111100000000000 2014 02 02 06 31 18 : K22: D001010100011111001110 DECABIT: CH1:50-ON RAW: 11111000010000000000000000000000011111111111000000000001111111111100000000000111111111110000000000000000000000000000000001111111111111111111111111111111111111111111111111111111000000000000000000000011111111111111111111111111111111100000000000 2014 02 02 06 35 29 : K22: D001010000011111001110 DECABIT: CH1:50-OFF RAW: 10000111101000000000000000000000011111111111000000000001111111111100000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111000000000000000000000011111111111111111111111111111111100000000000 2014 02 02 06 55 16 : K22: D001010000011111001110 DECABIT: CH1:50-OFF RAW: 10000111101000000000000000000000011111111111000000000001111111111100000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111111111111111111000000000000000000000011111111111111111111111111111111100000000000 ^K22-Channel 8 + Decabit Command-50 = Street Lighting
It seems that decabit commands can be sent in any K22 slot to have the same effect. The decabit decoder would look only for a valid start bit and bit-sequence. The older K22 would run though a 3 minute cycle and only look at the channels it was programmed to listen for. As a result, you will see the commands being packed together (like, at 16:55:15) to turn everything off in a 3-minute transmission cycle, and other times, you will find a whole 3-minute cycle is run just to send one decabit command as above, while keeping the K22 receivers at their appropriate values.
Most of the K22 channels are enabled progressively starting from 21:55:15 where they are all enabled by 1:25:15 and then all disabled again at 6:55:15. They get progressively re-enabled by 8:55:15 and are all up by 11:55:15 and then progressively disabled until all turned off at 16:55:15. The decabit channels seem to follow similar patterns, with the cycle repeating daily.
I suppose this information would easily let you track down whether your acoustic noise and flicker problems are ripple signalling related. In order to fix this, apparently, you will need to invest in fairly expensive filtering on a per-appliance basis (or even larger filters for whole-residence basis).