Problem: Opal Top-Up Pending Collection – RIP, my Opal Card

It’s probably self-evident by now, as a regular user of public transport (and an observant one at that), that I’ve been looking forward to receiving and getting onto the Opal system. While concession holders, including tertiary students, were left to last in the phased roll-out, UNSW was fortunate to be one of the first to introduce the concession Opal to its students amid privacy fears.

20150311_094725Of course, it was all a storm in a teacup – we really had little to no choice. Most of us had to share our data sooner or later, as paper concession tickets were now on a timeline towards death – possibly as soon as the end of the year. Part of that includes the disappearance of these “pre-boarding” validators for the UNSW bus queue. These rechargeable, battery operated units on hand-trolleys utilized the same Datafare 2000 console and two AES Prodata ticket readers for magnetic stripe tickets. These stamped the route number as U11, and were originally introduced in pairs (four readers) to speed up boarding of the large number of passengers to and from UNSW during peak periods.

20150417_115401With the new Opal card, pre-validation was no longer possible or required, as the processing time per user drops dramatically with the NFC card based system. Instead, we now use front and rear door boarding, for an equivalent of four readers per bus (when all operating) and approximately double to triple throughput at an estimate.

Just a few weeks ago, I was very pleased with myself when I finally used up the last ride on my last myBus2 Travelten Concession ticket. Good riddance to paper tickets, to waste, and to slow boarding.

 

The Honeymoon Period

I did use the card for as many transactions as I could, as avoiding a ticket queue, waiting for a pinpad to process the transaction or fumbling for change is something I’d rather not do if I could avoid it. I did have some paper tickets remaining, so I tried to use them up as soon as possible.

Opal-Success-Topup

I even topped up with $20 via the online system just so I could keep using it. No dramas, at least, on the first time. Easy enough to use, I didn’t have to think about it anymore. But I didn’t trust it enough to do auto top-up just yet. I mean, why should I expose myself to any more potential risk – manual pre-paid top-up works fine for me on my mobile phones, so I’ll keep it that way on my card.

I enjoyed receiving the benefits – for example, now I get concession discount off off-peak train fares, unlike before. The 60-minute transfer window also makes bus-journey-breakage to visit a computer shop and go home, or the post office, much more economical – sometimes resulting in fares below half the price I used to pay when doing two separate single tickets.

As my usage on Opal became heavier, my balance started to approach the point of nervousness – did I have enough to last me another day? So, as a responsible user, I decided to give the top-up another whirl on Saturday, this time opting for the maximum AU$60 top-up, to prepare the card for another week of travelling.

Opal-Top-Up-Receipt

The top-up was committed to their web interface on the 18th April, and checking my bank’s statement, the cash was successfully taken on the 20th April.

MoneyTaken

Of course, the online system showed a $60 top-up pending collection, which should mean that it would be applied to the card at the next tap-on at a reader. Jolly good. My next travel appointment was on Tuesday, so I’ll be all set!

Or so I thought …

The Honeymoon is Over

On Tuesday (several days after the online-top-up), when I began my journey, tapping on did not actually show “Top Up” and the new balance, as it did the last time when I did an online top-up. Instead, I had my old balance and the regular “ding” green-screen indicating successful tap-on.

I whipped out my phone … and checked the balance. Top-Up still pending. Hmm. I travelled the day, watching my balance continue its downward spiral towards zero. My only thoughts were … please don’t die on me Opal! I need you to travel!

Screenshot_2015-04-21-11-45-43 Screenshot_2015-04-21-16-50-28

The next time you present your card at a reader? But I have! Numerous times! And your system knows it too – just see the balance …

Waiting wasn’t doing the card any favours, so I decided to bite the bullet and seek help.

Opal Customer Care

There are two main ways to receive support for your Opal card, and none of them involve Sydney Trains staff – after all, Opal is run by an external party. The two methods are via the internet (i.e. by using their support form from the logged-in section on Opal.com.au) or by phoning the 13 OPAL (13 67 25) phone number, at the cost of a local call (or more if on mobile, and depending on your carrier).

When I reached the university on Tuesday, I sent them a message by the website. Sadly, I heard nothing from them even into the evening when I finally got home. Since 13 OPAL is staffed 24/7, I decided to call them up and see what they had to say.

I generally don’t prefer calling a company purely because I have to sit on the line and listen, or wait on hold, or key in many digits and navigate phone-trees. Opal is no different in this regard, but since I have my VoIP capture system set-up (unfortunately, not immune to packet loss), you can “listen in” on the journey to get to an operator:

  • The Call is Answered – Greeting Message
  • “Thank you for calling Opal Customer Care. To activate your Opal card, press 1. To learn more about the Gold Senior Pensioner Opal Card, or to apply, please press 2. To learn more about the Concession Opal Card, or to apply, press 3. For all other enquiries, press 5.”
    • Note: There is no option 4! I press 5.
  • Opal Card Number Query
  • “If you have your Opal card number, press 1. If you do not have an Opal card number, press 2.”
    • I press 1.
  • Enter Opal Card Number
  • “Please enter your Opal card number – it’s the 16 digit number located on the back of your card.”
    • I enter all 16 digits … making sure not to get it wrong.
  • Confirm the Opal Card Number
  • “Let me confirm the Opal card number you entered. {Zero, Two, Three, Five, Six, Eight}. If that’s correct, press 1. If not, press 2.”
    • Note: I have modified the Opal card number readback for one sample of each unique digit in my Opal card. Missing are samples for 1, 4, 7, 9. I press 1.
  • Enter PIN
  • “Please enter your PIN. It’s the four digit number you provided when you registered your Opal card.”
    • I got a bit stumped, entered in the wrong pin …
  • Wrong PIN
  • “Sorry. That PIN doesn’t match my records. [Pause] Let’s try again.”
    • It redirects to the Enter PIN phrase, and this time I get it right.
  • Main Menu
  • “Main menu. To hear your balance, press 1. To add value to your card, press 2. To hear recent transactions, press 3. To set up auto-top-up, press 4. For more information about Opal, press 5. At any time, you can use the following keys to navigate through the options. Press * to repeat the menu. Press 7 to return to the previous menu. Press 9 to return to the main menu. Press 0 to transfer to a customer service representative.”
    • At long last, we finally get a way to transfer out to a representative. I press 0.
  • Transfer Message
  • “Thank you for calling Opal Customer Care. You are now being transferred to a customer service representative for assistance.”
    • You don’t have to do anything here, but the first time I called, this message failed to play resulting in an awkward silence before something finally happened.
  • Opal Customer Care Queue and Answers
  • [Ring] Thank you for calling Opal Customer Care. Your call is important to us and will be answered by the first available customer service represent … [Call gets answered, name censored out].”

Guess what the first thing they asked me for after describing my problem? Yes, that’s right. They wanted my Opal card number … again. Then they verified all my personal details before proceeding.

I must say, the procedure to get to the representative is relatively longwinded and took several minutes. However, I never had to wait on hold for an appreciable amount of time, and the call was answered by a courteous representative who was understanding and clear with their instructions. The call was relatively brief and answered my questions.

Sadly, it was only after the call to them that they finally responded in writing to the same effect as what was said over the phone. Then they closed the ticket immediately. It seems their online support system is lacking severely as you can’t review your submitted messages or respond to tickets online via the logged-in area of Opal.com.au.

Opal-Reply

Lets try some Self-Help

As per the instructions provided above, the first method of self-help is to tap on at a train station, and then tap off after a short delay to force a fee-free tap-on reversal that will update the card status.

I tried this twice, once at Auburn when I managed to trigger the immediate-tap-off protection on the barrier thus resulting in a “Please wait to tap off” message. After waiting a minute, I was allowed out, and then tapped back on to try and continue my journey. No dice, the balance is not updated.

I tried again at Central, this time, waiting an appropriate time to avoid triggering the error message. Still no dice.

Opal-Activity

Scanned DocumentMy last resort was to do a “small” in person top-up at a bricks and mortar retailer. I was informed that this would consolidate the pending top-up with the top up applied in person and write it onto the card immediately.

I walked around Central station and found a top-up machine next to a few signs advertising its availability.

20150423_103333

Earlier last month, I spotted crews installing the machines but I didn’t really have a reason to post the image until now. Anyway, a lady was fumbling with the machine and couldn’t work out what to do with it.

20150319_110829

As a result, I visited a shop just next to the station. My balance was dire – just $1.59 at the time. The $20 I paid in cash was added to the card, but that was all. My $60 didn’t turn up and remained pending online. Upon further examination of the receipt, it seems there is some anomalies – notably the Transaction Number showed just 36 – according to the online log, the top-up took place around transaction 102. The card number was also identified but missing the last digit (seems likely the last digit may be a useless check-digit or this may be an ePay terminal special).

At least I could continue travelling, but my $60 is missing … in the ether.

Nope. That Didn’t Work!

I sent them a new online message, a little earlier than before, on Thursday thinking that might give them a chance to answer the query and I wouldn’t have to call in. Sadly, I was wrong, and no answer was forthcoming even into the late evening. I decided to call them, and owing to downtime with my main VSP, I had to go to a backup VSP to complete my call.

Sadly, there wasn’t any way out of this short of blocking the old card as a defective card (i.e. what they do when it’s reported lost/stolen) and reissuing a new card by post which requires activation, with the consolidated balance on the new card. As a result, I will be out of an Opal card for a few days, even with their promise to dispatch it by express. After finalizing the call, the e-mail started coming in …

Opal-Blocked

Opal-OrderConf

It was only this morning (Friday) that they finally answered my online query – noting that I had called them up. In short, if you need something fixed, call them up. Their online support is just poor by comparison, despite their insistence that the queries will be faster “because they have all your information”.

Opal-FinalReply

As a result, my Opal card is now useless to me, and not valid for any form of transport. [Insert Sad Face]

OhnoCardFront

RIP, my first concession Opal card. 3085 xxxx xxxx 3052, a Mifare DESFire EV1 with UID 0449xxxxxx2c80 you will be missed.

Some Observations of the Opal System

By looking at how the system operates, as an active user of the system, I can make some educated guesses as to how the system is implemented and how it “works”. In short, it is quite a complex system in order to meet everyone’s needs.

20150319_170152

The Opal system seems to consist of a few different sorts of readers – the “standalone” sorts that sit on Opal poles at smaller stations, the “barrier integrated” type at larger stations, the “bus” readers which are configured to work with a drivers’ console, and possibly even others that I haven’t encountered due to a lack of “ferry” usage.

The readers themselves are fairly complex pieces of equipment – of course, they have NFC and a colour LCD, but they also seem to have more intelligence in them. I suspect the bus units have a full knowledge of stops, and distances between stops, and rely on the drivers’ console to inform the units of the run number, stop location (GPS or manually selected) to enable a tap-on or tap off. The train readers probably have similar data for the train network distances as well. Each of the readers also has to have knowledge of the time, likely via NTP over network as they all seem to have an IP connection of sorts (as evidenced by the image of the booting reader above). I suspect bus readers, being mobile, rely on some form of 3G/4G connectivity which explains the delays experienced with updating Opal journey data and top-up data.

The readers also have to take on additional duties – aside from just reading cards, deducting fares, writing back to card, the Opal system seems to be a “hybrid” system where the value is stored both on the card and on their servers.

The primary source of data is generally the card, which stores the remaining value on the card, the tap-on data (i.e. location, time), as well as details of the last trip (when last tapped off, what mode, distance at the least) so as to work out whether it’s a transfer or not. As I guess the readers may be NTP synchronized, the time stamp drift between readers may be in the order of tens of seconds – this might result in the time recorded on an Opal card to be more granular (say, minutewise), resulting in the “Please wait to tap off” message on attempting an immediate tap-off cancel. This type of operation allows the card to continue working and to hold balance even in a temporary outage of the back-end or in case of connectivity issues, which is its advantage.

However, the readers must also log all their changes in a journal and upload this periodically back to the back-end so that the online journey log and balance can be updated. This is done relatively frequently (in a matter of minutes) in most cases, so both online and card are kept in sync. In the case of bugs or system failures, then it is possible that things will fall out of sync, and permanently so if data is lost.

The readers must also accept a differences log to apply to new cards they see. All new cards are issued unactivated with no balance written onto the card, and must be presented to a reader within 30/60 days to “collect” the top-up. What this means is that upon receiving your card and activating your card online, Opal must send out a record to all the readers that if it spots your new card tapping on, it needs to apply a credit of the amount you paid when signing up for the card. This allows the credit to remain safe even if the cards are lost in transit.

The differences log also extends to online top-ups, and likely for auto-top-ups as well. This is also probably what allows for the blocked card feature to protect the balances.

As with all databases, the end units will have a limitation as to the number of difference records they can handle purely because of space limitations in storage and processing time limitations when tapping on to “scan” the database for your card’s UID and apply any needed changes, and write it back, all in under one second.

I believe the overflowing of this differences log is the cause of initial top-up delays, which the representative informed me were indeed occurring to varying degrees. This also explains the reasoning behind the phased introduction of the Opal system – otherwise this differences log would be overflowed to the point of system collapse as new cards fail to come online or top up fails to be applied.

By now, you’re probably wondering why they didn’t opt for an “online” transaction system. In fact, it could be possible to have an online system if the barriers were at fixed stations, connected by fibre, backed-up with a very powerful cluster of database servers, as they can process a transaction in 100ms or less, which would be imperceptible. However, to ensure the safety of the data, adding crypto to this may increase the system load and latency slightly. But such an online system becomes vulnerable to loss of connectivity and is not suitable for 3G “mobile” stations due to round trip latency, or retrofitted Opal poles where the only thing available may only be power.

Because the card seems to be the primary source of the data, authorized payment terminals can make alterations to the card data to inject balance onto the card for immediate use – even if it didn’t show up on the back-end’s logs. This appears to be by design, to avoid any requirements on retailers to have a constant internet connection, and to avoid total system collapse if the online-pending-updates continues to cause trouble. I suspect if the top-up data isn’t eventually synchronized, auditing the system for “money leaks” might become difficult.

However, something seems to have gone fishy with my card. The card itself is an NXP MiFare DESFire EV1, which is known to be a secure cryptographic NFC card with special features intended for fare collection systems including methods of ensuring data integrity even in varying field conditions (i.e. card removed before write complete). Unless the card isn’t being used with the right data types and programming order, corruption of the card is extremely unlikely but possible.

The first clue to something fishy about the card comes from the manual top-up where the transaction number doesn’t seem to match. I haven’t done a manual top-up before, so I don’t know if this is by design, but a mismatched number alludes to my other hypothesis about why the card failed. The number 36 would correlate to travel on 18th March, a month ago.

I have a suspicion that the card readers are updated with a differences log for top-ups that includes not only the adjustment required to the card, but also the transaction number it should be taking place at. This allows for the transaction to be done unambiguously in a sequenced manner. However, when I tapped on after my top-up, it may have been during a delay in data or the reader itself was operating off stale data resulting in the expected state of the card (transaction number wise) being different to that of the differences log after my first bus ride.

From then on, all the readers may have refused to apply the adjustment noting that the card status, transaction number wise, is above the expected number where the top up should be applied, in order to avoid double-top-up of the card in case the successful top-up from the previous reader is not yet sent back to the database for clearance from all readers.

As a result, the top-up is “caught in limbo” – readers won’t apply it because the card doesn’t contain the status they expect it to, and the online database system doesn’t seem to update the difference log to “inform” readers to try it again.

Of course, I wasn’t involved in developing the Opal system, and I don’t have any connections to the insides, so I can only surmise its operation based on its observed behaviour. However, it’s hardly encouraging to note that my second top up resulted in the death of a card.

A second possibility is that the card indeed faulted during a write operation and corrupted some non-critical data (transaction count?) which was then “built upon” by following readers. The possible clue to this is when ticket barriers tell you to “Try again with only one card”, or tell you “Invalid”, or error 90, 91, 92, but a retry succeeds. I suspect the cause of this may be due to timing problems, but I’ve always pulled my card out for use with the reader and placed it as close as possible to the target, as I know that an “on air collision”, while designed against in the IEC standard, is a possibility, as is a momentary loss of field in the case of “waving” or “tapping” a card against a reader (as some misinformed passengers seem to do, to bemused faces when it doesn’t work). I don’t expect this to be a major issue though, as the cards were designed against these problems if used according to NXP’s recommendations, and the worst that should happen is that a write fails and the card maintains the previous state.

Conclusion

The Opal card system, having all card types now available, with further sign-ups expected for Concessions through the end of the year, seems to have been a successful rollout in the eyes of the media. That doesn’t, however, mean that it is bug free – which is not necessarily a surprise but comes rather as an annoyance.

If you end up getting “trapped” with an online top-up that you can’t collect, the only solution (as I have been informed) is to have the card replaced which leaves you several days out of a card with no compensation whatsoever. This is hardly the convenience we “signed up” for, and really shouldn’t be happening. Apparently, according to the representative I spoke with, it doesn’t happen “often”.

The procedure is basically the same as blocking a lost card, and transferring the balance to a new card. I think this may have been misdiagnosed as a defective card, as it’s probably the easiest way out of such a problem, but technologically speaking, the data on the card can be repaired.

It seems likely that an inconsistency occurred somewhere along the lines resulting in a failure of the top-up difference data to apply to the card, possibly due to the card being used with a reader with outdated difference data, or having a “sheared” write (which can be avoided with proper use of data-types and write procedures on the MiFare card system) resulting in a corrupted transaction counter on the card. The fact the card can continue to be written and read to, with manual top-up, suggests to me that any failed write did not damage the permissions of the MiFare blocks, which would otherwise reject a read/write request causing the card to be permanently bricked.

The fact that the data is actually synchronized with a back-end database implies to me that it would not be impossible to repair a card based on an agreed synchronized data (if the permissions are not broken), possibly by implementing a new “differences log” mode on the readers to overwrite the card data completely with synthesized data from the online database at the next tap on based on a manual request. However, they might be hesitant to do that for several reasons – as it seems the readers may have too many difference log data to deal with, and it creates another potential avenue for disputes or tampering if the synchronization occurs incorrectly. That being said, allowing retailers to physically top-up cards does cause some security risks as well, but they seem to have no problem with it.

Regardless, I’m out of an Opal card for at least a weekend, so that means no travelling unless I want to revert back to paper and forego any Opal discounts to which I am entitled to. Thanks Opal. If you have trouble with your Opal card, do make sure to call them on 13 OPAL, as their online turn-around times are not particularly great.

About lui_gough

I'm a bit of a nut for electronics, computing, photography, radio, satellite and other technical hobbies. Click for more about me!
This entry was posted in Computing, Travel and tagged , , , , , . Bookmark the permalink.

7 Responses to Problem: Opal Top-Up Pending Collection – RIP, my Opal Card

  1. sparcie says:

    I haven’t travelled on a bus/train in a looong time, but I’m curious. Will the opal system eventually completely replace cash and paper tickets? If I were to want to travel on a bus or train as a one off would I need to get an opal card to do so? Otherwise I think it looks like a great system, just I don’t have any personal experience with it!

    Cheers
    Sparcie

    • lui_gough says:

      I think the plan is that the Opal system doesn’t replace the paper tickets entirely at this stage. At least, not until they introduce a single ride disposable Opal ticket – which is possible as there are MiFare Ultralight paper-NFC cards that cost barely a few cents to produce each, and are “disposable”. Until such time, they will maintain single tickets for adults, but leave the enticement of “lower fares” through the Opal system as an incentive to make you want to switch.

      From a technical perspective, they have all the reason to get rid of magstripe tickets – they’re very problematic in terms of head cleaning/alignment and mechanical maintenance of readers and dot matrix / thermal print heads. The miles of cardboard stock they use, and the bins in the barriers that need manual attention, along with the occasional “the machine ate my ticket” are all costs to staff which the Opal system can entirely reduce or remove.

      In some overseas systems, single/disposable tickets in the form of NFC cards are already common (e.g. Hong Kong MTR Airport Express Tourist Pass), whereas others use flexible plastic cards with NFC technology that are forcibly collected by the barrier at the end of a single trip (e.g. Shanghai Metro) to reduce waste (as they are all reusable to some extent).

      – Gough

  2. Pingback: Oh No! Another Opal Card, but Opal Activation is Broken! | Gough's Tech Zone

  3. Pingback: The tarnished Opal continues to darken … | Gough's Tech Zone

  4. Myrle Bowden says:

    Hi, we have almost the same situation. I usually topped up online via my OPAL account $120.00 fortnightly for my trips to and from work. My problem started January 15, 2016. The $120.00 is still floating around. For 2 weeks I’ve been waiting for the $120.00. I’ve been told it may take another 5 days or less before the $120.00 will be returned to my bank. So in the mean time I have been feeding my OPAL in cash. As I write in your post, I am still waiting for the $120.00.

  5. amanda says:

    Same problem. Did auto top up and they kept taking my money but not topping me up. Took $10 off me 3 times and still havent recived it on my top up card!! Opal is shit

  6. Kate says:

    This happened to me last year, and I topped up my card with cash, expecting to have to replace the card. Funny thing is, I managed to ‘collect’ the top up at some point, and top ups online have been successful. Until now. It’s happened again :/ They should really fix this problem.

Error: Comment is Missing!