Checking it out: Cityrail Real-Time Train Information

It seems like a miracle of our state government – in such a short space of time, public transport services are getting more convenient than ever and the data collection and access is seemingly no longer a problem. The modernization of timetables into phone applications by third parties helped quite a bit in making it easier to navigate while providing free versions for the benefit of the general public – thanks Grofsoft!

We already know that live bus data is available, but this has now extended to train data as well for selected lines. Access to this train data was apparently granted to a limited number competing developers resulting in six apps being live-train-data enabled:

  • TripGo by Skedgo – available free on iOS5.1+ and Android.
  • TripTastic – available for iOS6+ only at $2.99 at time of writing.
  • Tripview by Grofsoft – lite (ad supported) version available free with minor limitations on iOS4.3+ (as iPhone app) and Android, full version $2.99 on iOS App Store and $3.29 on Google Play at time of writing.
  • Arrivo Sydney by TripMate – lite (ad supported) version available free on Android, full version $2.99 on Google Play at time of writing.
  • Hidden City by Small Potion – available free on iOS5+ and Android.
  • TransitTimes by Zervaas Enterprises – free edition available on Android with strict 1 search per hour limitation, only paid edition available for iOS5+, full edition $2.99 on iOS App Store $2.99 on Google Play at time of writing.

Live train data currently covers “Western, Bankstown, Inner West, Northern, Cumberland, Airport and East Hills, South and North Shore Lines as well as the Eastern Suburbs and Illawarra Line to Heathcote.”

Already, looking at the apps, we can see that Tripview is a bit old fashioned in being an iPhone style app – but it also offers the widest iOS compatibility for those still rocking the older iPhones. Hidden City is a bit of a special app – as it’s less oriented towards actually searching times, but instead is designed to plan mystery trips with the aid of public transport. TransitTimes strangely is only available as paid on iOS with a free but quite limited edition on Android – whereas Triptastic has a paid iOS app only.

So, lets take a peek at how it works and what the data looks like.

Methodology

A computer with two NICs is set to bridge the two NICs at the layer 3 level (and is thus transparent to the devices). On one port of the bridge, a connection to a NAT’d internet connection is provided. On the other port, a wireless router is attached with NAT disabled, operating in AP mode. The wireless tablet devices are joined to this AP, and all their packets are captured by Wireshark as they traverse the bridge. Classic MITM.

Free apps – Tripview Lite and TripGo were installed on an iPad (original 1st generation running iOS 5.1.1) and various lookups were initiated using the interface of the app. Arrivo and TransitTimes were tested on an Asus Nexus 7 (Wi-Fi only, 16Gb model) running Android 4.2.2.

Hidden City was not tested as it’s more oriented towards city discovery through planning mystery trips – its behaviour is a bit hard to predict and depends on several variables – such as when you are searching, how much time you have available and what your present location is.

The resulting data from Wireshark is interpreted and provided in this report.

Of course, all of this work is done out of personal interest as an Engineer and is in no way intended to cause trouble to the individuals and companies running these services. Information is always interesting – someone might find new and exciting ways to make use of this information which we cannot get directly from the government.

Results – Grofsoft Tripview Lite

Lets start with Tripview – one of my favourite apps and one that I know well already. Since I’ve had it running for a while – the initial timetable downloads have been done – you can always look at my previous article about live bus times if you’re interested in how the timetable request is made – but the files are small, less than 2Mb a piece! That’s quite an achievement.

Upon starting up the application, it contacts Grofsoft to check whether there are updates to the timetables and service information. The request URIs and the responses are:

type=timetable
name=syd-trains
time=1365735537
size=149613
version=15

type=timetable
name=syd-ferries
time=1365294296
size=8060
version=15

type=timetable
name=syd-buses
time=1365294420
size=1167405
version=15

type=config
adp=2
hp=2
rtbo=SB,CR
adpri=3,2,1
type=news
start=1365747454
end=1365747454
title=131500 Trains
body=Passengers on #SouthCoastLine btn Kiama and Bomaderry (Nowra), allow extra 10mins travel time due to a train requiring mechanical repairs.
url=http://mobile.twitter.com/131500trains

type=news
start=1365745871
end=1365745871
title=131500 Trains
body=Express buses will be running between Central and Randwick Racecourse for Australian Derby Day tomorrow - details http://t.co/8lO7YMbGbM
url=http://mobile.twitter.com/131500trains

type=news
start=1365745725
end=1365745725
title=131500 Trains
body=Shuttle buses running btn Central and Moore Park from 5pm for tonight's Roosters v Bulldogs game at Allianz Stadium http://t.co/mmS1fJ0FSh
url=http://mobile.twitter.com/131500trains

[... SNIPPED ...]
type=trackwork
timetable=syd-trains
start=1365782400
end=1365955200
lines=il,CR_il
title=Eastern Suburbs & Illawarra Line - Trackwork
body=There is no trackwork on the Eastern Suburbs & Illawarra Line; however, trackwork on the East Hills Line may affect how you travel.
url=http://www.grofsoft.com//tripview/svinfo.php?trackwork=D8B9B02BE0D35CDDE044002128FEC996

type=trackwork
timetable=syd-trains
start=1366387200
end=1366560000
lines=il,CR_il
title=Eastern Suburbs & Illawarra Line - Trackwork
body=There is no trackwork on the Eastern Suburbs & Illawarra Line; however trackwork on the City Circle and Airport lines may affect how you travel.
url=http://www.grofsoft.com//tripview/svinfo.php?trackwork=D958C34E68DE65D2E044002128FEC996

type=trackwork
timetable=syd-trains
start=1366632000
end=1366644600
lines=il,CR_il
title=Eastern Suburbs & Illawarra Line - Trackwork
body=Buses replace trains between Bondi Junction and Central.
url=http://www.grofsoft.com//tripview/svinfo.php?trackwork=D9E783C18C4163D2E044002128FEC996

[... SNIPPED ...]

Also note, they may be checking the User Agent – last time with the buses, I noted they were returning blank tuples for incorrect user agents – so get your user agent switcher out. The user agent of the app is currently:

TripViewLite/223 CFNetwork/548.1.4 Darwin/11.0.0

I will be focusing on the live train data part of the app – the first thing we must do is search a trip – lets take Central to Granville as an example trip. Confirming it and getting the list of times, a request for live train data is sent URL-encoded as:

{
  "timestamp" : 1365817423,
  "pollInterval" : 30,
  "delays" : [
{"route":"CR_bk_u","tripId":"684F.1203.102.130.A.8","stopId":"2170822","start":"11:35","offsets":"11:35,0"},
{"route":"CR_bk_u","tripId":"710J.1203.102.130.A.8","stopId":"214381","start":"10:44","offsets":"10:16,0,11:21,1,11:22,0"},
{"route":"CR_bk_u","tripId":"657G.1203.102.130.A.8","stopId":"2170822","start":"12:05","offsets":"12:05,0"},
{"route":"CR_bk_u","tripId":"658E.1203.102.130.A.8","stopId":"2170822","start":"11:05","offsets":"11:39,0"},
{"route":"CR_bk_u","tripId":"655G.1203.102.130.A.8","stopId":"2170822","start":"10:33","offsets":"10:33,0,11:36,1,11:37,0"},
{"route":"CR_bk_u","tripId":"682J.1203.102.130.A.8","stopId":"214381","start":"11:14","offsets":"10:43,1,10:46,2,10:47,1,10:49,2,10:51,1,10:52,2,10:56,3,10:59,2,11:01,3,11:03,4,11:04,3,11:08,2,11:14,3,11:15,2,11:16,3,11:18,2,11:19,1,11:23,0"},
{"route":"CR_bk_u","tripId":"685G.1203.102.130.A.8","stopId":"214381","start":"11:44","offsets":"11:16,0"},
{"route":"CR_bk_u","tripId":"702H.1203.102.130.C.8","stopId":"2170822","start":"10:02","offsets":"11:25,0,11:33,1,11:36,0"},
{"route":"CR_bm_d","tripId":"W533.1203.102.130.V.4","stopId":"2000332","start":"09:48","offsets":"10:34,0,10:37,1,10:38,2,10:40,3,10:41,2,10:43,3,10:46,1,10:47,2,10:51,3,10:52,2,10:55,3,10:58,2,11:03,1,11:06,2,11:08,1,11:12,0,11:14,1,11:18,2,11:20,1,11:25,0,11:26,1,11:28,0"},
{"route":"CR_bm_d","tripId":"W537.1203.102.130.V.4","stopId":"2000327","start":"11:18","offsets":"11:18,1,11:21,2,11:27,1,11:36,0"},
{"route":"CR_bm_d","tripId":"W535.1203.102.130.V.4","stopId":"2000327","start":"10:18","offsets":"11:03,0,11:04,1,11:05,0,11:06,1,11:07,0,11:08,1,11:13,2,11:16,0,11:17,1,11:21,2,11:22,1,11:25,2,11:28,1,11:30,2,11:33,1,11:38,0,11:40,1,11:42,0"},
{"route":"CR_bm_d","tripId":"W541.1203.102.130.V.4","stopId":"2000327","start":"12:18","offsets":"12:18,0"},
{"route":"CR_bm_d","tripId":"W531.1203.102.130.V.4","stopId":"2000327","start":"09:18","offsets":"10:04,0,10:13,1,10:16,0,10:25,1,10:26,0,11:15,1,11:17,0,11:24,1,11:27,0,11:38,5"},
{"route":"CR_st_d","tripId":"651G.1203.102.130.S.8","stopId":"2000341","start":"11:10","offsets":"10:51,0,10:52,2,10:55,0,11:29,1,11:31,0"},
{"route":"CR_st_d","tripId":"706J.1203.102.130.S.8","stopId":"2000341","start":"10:56","offsets":"10:37,0,10:38,2,10:41,0,10:57,1,10:58,0,11:15,1,11:17,0,11:24,1,11:25,0"},
{"route":"CR_st_d","tripId":"677F.1203.102.130.A.8","stopId":"2000341","start":"11:26","offsets":"11:27,0"},
{"route":"CR_st_d","tripId":"654H.1203.102.130.S.8","stopId":"2000341","start":"11:39","offsets":"11:20,0,11:21,2,11:24,0"},
{"route":"CR_st_d","tripId":"707E.1203.102.130.A.8","stopId":"2000341","start":"11:56","offsets":"11:37,0,11:38,2,11:41,0"},
{"route":"CR_st_d","tripId":"653G.1203.102.130.S.8","stopId":"2000341","start":"10:40","offsets":"10:21,0,10:22,2,10:25,0,10:43,1,10:45,0,10:59,1,11:01,0"},
{"route":"CR_st_d","tripId":"55-H.1203.102.130.C.8","stopId":"2000341","start":"12:10","offsets":"11:51,0"},
{"route":"CR_wt_d","tripId":"100F.1203.102.130.T.8","stopId":"2000338","start":"09:51","offsets":"10:50,53"},
{"route":"CR_wt_d","tripId":"103F.1203.102.130.T.8","stopId":"2000338","start":"11:06","offsets":"10:36,0,11:00,1,11:02,0,11:08,1,11:10,0,11:27,1,11:28,0"},
{"route":"CR_wt_d","tripId":"119F.1203.102.130.T.8","stopId":"2000338","start":"11:36","offsets":"11:36,0"},
{"route":"CR_wt_d","tripId":"157F.1203.102.130.K.8","stopId":"2000338","start":"11:21","offsets":"10:49,5,10:50,4,10:51,9,10:56,10,10:58,9,11:01,10,11:04,9,11:07,8,11:11,7,11:12,8,11:15,9,11:17,7,11:19,6,11:21,7,11:22,6,11:23,7,11:25,6,11:31,5,11:32,6,11:33,5,11:35,6,11:36,5"},
{"route":"CR_wt_d","tripId":"117F.1203.102.130.T.8","stopId":"2000338","start":"10:36","offsets":"11:37,0"},
{"route":"CR_wt_d","tripId":"102F.1203.102.130.T.8","stopId":"2000338","start":"10:51","offsets":"10:23,0,10:45,1,10:47,0,11:08,2,11:09,3,11:11,4,11:12,3,11:13,4,11:14,3,11:20,2,11:21,1,11:30,0,11:32,1,11:39,0,11:41,1,11:44,0"},
{"route":"CR_wt_d","tripId":"101F.1203.102.130.T.8","stopId":"2000338","start":"10:21","offsets":"11:21,0"}],
  "vehicles" : [
{"route":"CR_bk_u","id":"1358.1359.1360.1361.1362.1363.1364.1365","tripId":"684F.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.884422,"lon":150.96103,"bearing":72,"lp":"Carramar:Villawood:bk:0.00"},
{"route":"CR_bk_u","id":"1534.1535.1536.1537.1538.1539.1540.1541","tripId":"685G.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.892887,"lon":151.02399,"bearing":195,"lp":"Birrong:Yagoona:bk:0.00"},
{"route":"CR_bk_u","id":"5256.5257.5258.5259.5260.5261.5262.5263-P","tripId":"702H.1203.102.130.C.8","dst":"Regents Park","sdst":"REG","shape":"T1_IWL_1","lat":-33.893208,"lon":151.14728,"bearing":293,"lp":"Lewisham:Summer Hill:iw:0.00"},
{"route":"CR_bk_u","id":"6486.6487.6488.6489.6490.6491.6492.6493","tripId":"655G.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.884113,"lon":151.20702,"bearing":22,"lp":"Central:Museum:bk:0.00"},
{"route":"CR_bk_u","id":"6910.6911.6912.6913.6914.6915.6916.6917","tripId":"657G.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.925552,"lon":150.9266,"bearing":39,"lp":"Liverpool:Liverpool:bk:0.00"},
{"route":"CR_bk_u","id":"7078.7079.7080.7081.7082.7083.7084.7085","tripId":"658E.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.919937,"lon":151.07593,"bearing":68,"lp":"Lakemba:Belmore:bk:0.00"},
{"route":"CR_bk_u","id":"9006.9007.9008.9009.9010.9011.9012.9013-P","tripId":"710J.1203.102.130.A.8","dst":"Ashfield","sdst":"ASH","shape":"T1_IWL_1","lat":-33.87892,"lon":151.20885,"bearing":187,"lp":"Town Hall:Central:iw:0.53"},
{"route":"CR_bk_u","id":"9926.9927.9928.9929.9930.9931.9932.9933","tripId":"682J.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_BL_2a","lat":-33.91394,"lon":151.15366,"bearing":108,"lp":"Marrickville:Sydenham:bk:0.00"},
{"route":"CR_bm_d","id":"6762.6763.6764.6765","tripId":"W537.1203.102.130.V.4","dst":"Mount Victoria","sdst":"MOU","shape":"T1_BML_1","lat":-33.819046,"lon":151.00699,"bearing":320,"lp":"Harris Park:Parramatta:bm:0.68"},
{"route":"CR_bm_d","id":"6802.6803.6804.6805","tripId":"W531.1203.102.130.V.4","dst":"Mount Victoria","sdst":"MOU","shape":"T1_BML_1","lat":-33.5951,"lon":150.2623,"bearing":321,"lp":"Blackheath:Mount Victoria:bm:0.82"},
{"route":"CR_bm_d","id":"6830.6831.6832.6833","tripId":"W533.1203.102.130.V.4","dst":"Mount Victoria","sdst":"MOU","shape":"T1_BML_1","lat":-33.712048,"lon":150.33092,"bearing":288,"lp":"Wentworth Falls:Leura:bm:0.99"},
{"route":"CR_bm_d","id":"6854.6855.6856.6857","tripId":"W535.1203.102.130.V.4","dst":"Lithgow","sdst":"LIT","shape":"T1_BML_1","lat":-33.69625,"lon":150.53522,"bearing":213,"lp":"Faulconbridge:Linden:bm:0.00"},
{"route":"CR_bm_d","id":"6890.6891.6892.6893","tripId":"W541.1203.102.130.V.4","dst":"Lithgow","sdst":"LIT","shape":"T1_BML_1","lat":-33.88345,"lon":151.20592,"bearing":213,"lp":"Central:Redfern:bm:0.00"},
{"route":"CR_st_d","id":"1166.1167.1168.1169.1170.1171.1172.1173","tripId":"677F.1203.102.130.A.8","dst":"Liverpool","sdst":"LIV","shape":"T1_SL_1","lat":-33.884792,"lon":151.20638,"bearing":217,"lp":"Central:Redfern:st:0.12"},
{"route":"CR_st_d","id":"1942.1943.1944.1945.1946.1947.1948.1949-N","tripId":"707E.1203.102.130.A.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_AEHL_2a","lat":-33.93578,"lon":151.1695,"bearing":74,"lp":"International Airport:Domestic Airport:eh:0.28"},
{"route":"CR_st_d","id":"3624.3625.3626.3627.3628.3629.3630.3631-N","tripId":"654H.1203.102.130.S.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_AEHL_2a","lat":-33.86713,"lon":151.21236,"bearing":26,"lp":"St James:Circular Quay:eh:0.32"},
{"route":"CR_st_d","id":"5096.5097.5098.5099.5100.5101.5102.5103-N","tripId":"55-H.1203.102.130.C.8","dst":"Circular Quay","sdst":"CIR","shape":"T1_AEHL_2a","lat":-33.929897,"lon":151.13997,"bearing":69,"lp":"Turrella:Wolli Creek:eh:0.00"},
{"route":"CR_st_d","id":"5662.5663.5664.5665.5666.5667.5668.5669","tripId":"653G.1203.102.130.S.8","dst":"Liverpool","sdst":"LIV","shape":"T1_SL_1","lat":-33.894962,"lon":150.93875,"bearing":189,"lp":"Cabramatta:Warwick Farm:st:0.00"},
{"route":"CR_st_d","id":"5758.5759.5760.5761.5762.5763.5764.5765","tripId":"651G.1203.102.130.S.8","dst":"Liverpool","sdst":"LIV","shape":"T1_SL_1","lat":-33.87164,"lon":151.0942,"bearing":311,"lp":"Strathfield:Homebush:st:0.00"},
{"route":"CR_st_d","id":"6302.6303.6304.6305.6306.6307.6308.6309","tripId":"706J.1203.102.130.S.8","dst":"Liverpool","sdst":"LIV","shape":"T1_SL_1","lat":-33.833035,"lon":151.01248,"bearing":308,"lp":"Granville:Merrylands:st:0.00"},
{"route":"CR_wt_d","id":"1324.1325.1326.1327.1328.1329.1330.1331","tripId":"157F.1203.102.130.K.8","dst":"Penrith","sdst":"PEN","shape":"T1_WL_1a","lat":-33.86386,"lon":151.06067,"bearing":266,"lp":"Flemington:Lidcombe:wt:0.37"},
{"route":"CR_wt_d","id":"7060.7061.7062.7063.7064.7065.7066.7067","tripId":"100F.1203.102.130.T.8","dst":"Penrith","sdst":"PEN","shape":"T1_WL_1a","lat":-33.76352,"lon":150.7837,"bearing":281,"lp":"Mount Druitt:St Marys:wt:0.90"},
{"route":"CR_wt_d","id":"7212.7213.7214.7215.7216.7217.7218.7219","tripId":"101F.1203.102.130.T.8","dst":"Penrith","sdst":"PEN","shape":"T1_WL_1a","lat":-33.75003,"lon":150.69595,"bearing":279,"lp":"Penrith:Penrith:wt:0.00"},
{"route":"CR_wt_d","id":"7380.7381.7382.7383.7384.7385.7386.7387","tripId":"102F.1203.102.130.T.8","dst":"Penrith","sdst":"PEN","shape":"T1_WL_1a","lat":-33.771515,"lon":150.84523,"bearing":253,"lp":"Rooty Hill:Mount Druitt:wt:0.00"},
{"route":"CR_wt_d","id":"7508.7509.7510.7511.7512.7513.7514.7515","tripId":"103F.1203.102.130.T.8","dst":"Richmond","sdst":"RIC","shape":"T1_WL_1b","lat":-33.801514,"lon":150.95651,"bearing":316,"lp":"Pendle Hill:Toongabbie:wt:0.00"},
{"route":"CR_wt_d","id":"9212.9213.9214.9215.9216.9217.9218.9219","tripId":"117F.1203.102.130.T.8","dst":"Richmond","sdst":"RIC","shape":"T1_WL_1b","lat":-33.650455,"lon":150.85109,"bearing":337,"lp":"Vineyard:Mulgrave:wt:0.00"},
{"route":"CR_wt_d","id":"9428.9429.9430.9431.9432.9433.9434.9435","tripId":"119F.1203.102.130.T.8","dst":"Richmond","sdst":"RIC","shape":"T1_WL_1b","lat":-33.891506,"lon":151.1421,"bearing":293,"lp":"Stanmore:Summer Hill:wt:0.78"}]

While the poll interval says 30 – it seems the app polls for the whole set of data every 20 seconds. The resulting information is quite large in terms of response, but credit to Grofsoft for implementing gzip encoding in order to compress the data and reduce the data transaction size to about 3kB per “ping” (or roughly under 1Mb/hour).

The naming of the routes seem to be quite self explanatory:

  • CR for Cityrail, followed by_ and  two letter “line” designator
  • bm for Blue Mountains
  • wt for Western
  • bk for Bankstown
  • st for South
  • ns for North Shore
  • nta for Northern (possibly North Strathfield to Hornsby)
  • ntb for Northern (possibly Epping to Chatswood link)
  • nc for North Coast
  • iw for Inner West
  • eh for East Hills
  • il for Illawarra
  • sc for South Coast
  • Followed by a _ and letter indicating run direction, u for up and d for down.

From this, it doesn’t seem it matters which station – the data is station agnostic, but the request is based on which lines your trip uses. So, in order to find out the coding for other lines, I did commit a Central to Hurstville, Central to International Airport, Cabramatta to Blacktown and Central to Hornsby trip, however, I couldn’t trigger the Cumberland line.

The response is broken up into two sections. The top table seems to describe the services and their running time history (possibly), whereas the bottom table describes the service and its present position in lat, long and heading. The “sdst” data item is used for the short destination name on the “network map” view. The “shape” data item probably commands the client to load a given train line shape highlight to project the future running direction of the service.

The tripId is a bit cryptic – it’s much longer than a regular run number – and the ID seems to be quite long as well and made out of many separated sequential numbers. Hmm. It also seems to show where it is “in between” two stations – but given this information, the implementation is a bit difficult as the application must interpolate, or do some statistical analysis of travel times to predict where the train will be, or how long the train will take to get to a given station. So, it’s not going to be 100% accurate, but if the update rate of the data feed is sufficiently high, the results can be quite good.

If one wants to get a better view of the network – you could craft a request like

http://realtime.grofsoft.com/tripview/realtime?routes=CR_bm_u,CR_bm_d,CR_wt_u,CR_wt_d,CR_bk_u,CR_bk_d,CR_st_u,CR_st_d,CR_ns_u,CR_ns_d,CR_nta_u,CR_nta_d,CR_ntb_u,CR_ntb_d,CR_nc_u,CR_nc_d,CR_iw_u,CR_iw_d,CR_eh_u,CR_eh_d,CR_il_u,CR_il_d,CR_sc_u,CR_sc_d&type=dv

which would provide both the up and down services on all the lines and at present, their server honours the request!

I’ve probably bugged the developers at Grofsoft enough by adding to their logs … but thank you Grofsoft! Your gzip implementation shows inspired consideration for your users!

Results – TripGo

Okay, so I started up TripGo before, so it may have downloaded some of the supporting data before I began my analysis. TripGo is a bit more confusing to me as a loyal Tripview user, but as an alternate data source of realtime information, it’s worth giving it some time.

Just selecting Central Stations and Departures, the number of packets flying by was astounding. Part of the reason is that it loads Google map images as we change views, zoom in and out – definitely eats a bit of data to do this.

The user agent reported by TripGo to http://gsp1.apple.com/pep/gcc (which seems to be a quick geolocation coding check) is:

TripGo/8546 CFNetwork/548.1.4 Darwin/11.0.0

The application makes several DNS requests – one is settings.crashlytics.com which it speaks TLS/SSL to and appears to be a crash analytics company. Another is tripgo.skedgo.com which is resolved to au.tripgo.skedgo.com at 123.100.229.37 which it also speaks TLS/SSL to. Unfortunately this means we can’t really get any data from this unless we go further and do an SSL MITM. Too much effort for too little reward I’d say, but at least, that secures the transport data and makes it (generally) impossible to know what trips you’re looking up. This security however, is broken since the Google maps requests are done in unencrypted HTTP so … hmm.

Results – Arrivo Sydney Lite

So, lets give Arrivo a spin and see what we can get out of it. Opening the program, we can see a DNS request for serv.tripmate.info resolving to 98.158.183.157.

The first thing it asks for is an update – the request is met with a 404 but interestingly Arrivo does not have a user agent – just the bare headers.

GET /syd_lite/2.0.1/update HTTP/1.1
Host: serv.tripmate.info
Connection: Keep-Alive

HTTP/1.1 404 Not Found
Date: Sat, 13 Apr 2013 02:32:29 GMT
Server: Apache
Content-Length: 219
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /syd_lite/2.0.1/update was not found on this server.</p>
</body></html>

Then it does something which no other client I’ve used does – it searches Twitter for tweets from 131500buses or trip_mate to provide the latest data.

GET /search.json?q=from:131500buses%20OR%20from:trip_mate HTTP/1.1
Host: search.twitter.com
Connection: Keep-Alive

Once that’s done, there’s a DNS request for data.tripmate.info resolving to 54.252.97.161 and then a TLS/SSL connection is established and we’re in the dark again. Damn.

Results – TransitTimes+ Free

Our final app to be tested for today.

On startup, the program does a DNS lookup for cdn.transittimesapp.com which points to a hefty cloudfront DNS to get the list of timetables. Interesting user agent, I must add.

GET /plus/1.0/version HTTP/1.1
Host: cdn.transittimesapp.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)

HTTP/1.0 200 OK
Content-Type: binary/octet-stream
Content-Length: 13534
Connection: keep-alive
Date: Sat, 13 Apr 2013 00:48:23 GMT
Cache-Control: max-age=3600
Last-Modified: Sat, 13 Apr 2013 00:01:13 GMT
ETag: "9c24276923bd6b781c1b907e300ce47e"
Accept-Ranges: bytes
Server: AmazonS3
Age: 3120
Via: 1.0 908de5488d3a0b3ebb974ab5e59786f8.cloudfront.net (CloudFront)
X-Cache: Hit from cloudfront
X-Amz-Cf-Id: QR2do3j1T5MD6I9j2elC-ZKuLKs0ceu3yXvXdrWhbbe9KfaHuWDxNg==

Then, it wants you to download the timetable – this is a hefty 23.57Mb download for Sydney at http://cdn.transittimesapp.com/plus/1.0/sydney/tt-25.zip. Not that much fun if you’re on 3G. Inside the ZIP file is a TransitTimes.sqlite database file unpacking to 82.5Mb and appears to be a GTFS format wrapped in SQLite. It’s also fairly chatty over the network too, because it seems to load from Google Maps as well.

Unfortunately, seeing live vehicle positions on a map turns out to be one of the things you have to upgrade for, so I’m not actually sure if the planned trip times are timetable only or whether they do have the benefit of live trip data. From what I can see, there were no requests for a live data stream – so the free app does not offer real time trains information from my experiments.

EDIT/UPDATE: I was contacted by the creator of the app which clarified that live positions are available by using Add Favourite -> Live Tracker (rather than Add Favourite -> New Trip). My bad! On the other hand, it’s always great to see developers looking out for opinion and experiences to improve their app. Hats off to Mr. Zervaas for taking the effort to clear this up.

The train live tracker appearance is quite nice, by selecting the line, you can see both directions at the same time. So instead of worrying about the trip, you can stare at the train icons and gauge how far away they are to work out the time, which is nifty when it’s the “should I walk a bit faster to the station, or take it slow” dilemma. Tapping on the train icons will give information on when the train’s position was last updated, which in some cases is several minutes. Of course, this is not the fault of the app – and is more likely to do with the data collection and distribution part of the live-positioning system (i.e. Department of Transport) – but at least we are told just how out of date the positions are instead of having to guess.

So, what’s happening on the data link you might ask? Well … possibly against the best wishes of the developer, but in line with my curiosity, it turns out that unencrypted JSON replies are used similarly to Grofsoft’s solution. The User Agent string is:

TransitTimes+ Free 7.0.1 for Android

And the request is for http://api.transittimesapp.com/rt/sydney/vp?c=SOU for the query of South Line trains. Replies are not encoded despite the requester stating the possibility of using gzip encoding which may be logistically due to increased server loading when compression is used.

GET /rt/sydney/vp?c=SOU HTTP/1.1
Accept-Encoding: gzip
User-Agent: TransitTimes+ Free 7.0.1 for Android
Host: api.transittimesapp.com
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Sun, 14 Apr 2013 01:12:07 GMT
Server: Apache/2.4.3 (Amazon) PHP/5.4.11
X-Powered-By: PHP/5.4.11
Content-Length: 2016
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json

The actual reply data looks like this:

{"r":[{"c":"SOU","p":[{"t":"659J.1203.102.130.M.8","ts":1365902571,"s":"Liverpool.S17.9 Loc","r":"SL_1","vi":"1022.1023.1024.1025.1026.1027.1028.1029","lng":150.960693359375,"l":"10:34 Circular Quay - Liverpool","lat":-33.87034606933594},{"t":"716G.1203.102.130.C.8","ts":1365902615,"s":"Granville.Clyde 4 Loc","r":"SL_2","vi":"2682.2683.2684.2685.2686.2687.2688.2689","lng":151.01705932617188,"l":"10:57 Liverpool - Circular Quay","lat":-33.8359489440918},{"t":"704J.1203.102.130.S.8","ts":1365902446,"s":"Liverpool.Liverpool 1 Loc","r":"SL_2","vi":"3704.3705.3706.3707.3708.3709.3710.3711","lng":150.92660522460938,"l":"11:43 Liverpool - Circular Quay","lat":-33.9255256652832},{"t":"653G.1203.102.130.S.8","ts":1365902607,"s":"Granville.S11.15 Entry Loc","r":"SL_1","vi":"3808.3809.3810.3811.3812.3813.3814.3815","lng":151.03689575195312,"l":"10:48 Circular Quay - Liverpool","lat":-33.854713439941406},{"t":"651G.1203.102.130.S.8","ts":1365902180,"s":"Sydney.0378","r":"SL_1","vi":"4192.4193.4194.4195.4196.4197.4198.4199","lng":151.2070770263672,"l":"11:18 Circular Quay - Liverpool","lat":-33.8841552734375},{"t":"706J.1203.102.130.S.8","ts":1365902619,"s":"Strathfield.L311","r":"SL_1","vi":"4384.4385.4386.4387.4388.4389.4390.4391","lng":151.15927124023438,"l":"11:04 Circular Quay - Liverpool","lat":-33.894161224365234},{"t":"705F.1203.102.130.A.8","ts":1365901481,"s":"Liverpool.Liverpool 3 Loc","r":"SL_2","vi":"4784.4785.4786.4787.4788.4789.4790.4791","lng":150.92660522460938,"l":"11:27 Liverpool - Circular Quay","lat":-33.92556381225586},{"t":"678J.1203.102.130.A.8","ts":1365902585,"s":"Liverpool.S17.6 Loc","r":"SL_2","vi":"7288.7289.7290.7291.7292.7293.7294.7295","lng":150.9622039794922,"l":"11:13 Liverpool - Circular Quay","lat":-33.86948776245117},{"t":"675E.1203.102.130.A.8","ts":1365902355,"s":"Strathfield.0120","r":"SL_2","vi":"7440.7441.7442.7443.7444.7445.7446.7447","lng":151.0941925048828,"l":"10:43 Liverpool - Circular Quay","lat":-33.871639251708984}]}]}

Strewth! This reply is a very long “line”, which I guess, makes sense in terms of reducing data transaction and needless characters since … well, to the app it wouldn’t make an iota of difference. The size of the reply is almost certainly smaller than the gzipped replies of Tripview. The information is queried every 15 seconds by the application. We see the presence of the same multiple-part run numbers, and GPS information. I guess it’s nice that the query is so simple in its construction – you literally can’t get it wrong.

So, lets try a quick dissection of the reply data (rough, based on my assumptions as a train traveller):

  • c = line – from the app:
  • AIR = Airport and East Hills
  • BLU = Blue Mountains
  • BNK = Bankstown
  • CAR = Carlingford
  • CMB = Cumberland
  • EAS = Eastern Suburbs and Illawarra
  • HUN = Hunter
  • INW = Inner West
  • NEW = Newcastle & Central Coast
  • NOR = Northern
  • NSH = North Shore
  • OLY = Olympic Park
  • SCO = South Coast
  • SHI = Southern Highlands
  • SOU = South
  • WES = Western
  • p = position data including:
  • t = tripID
  • ts = timestamp of data update
  • s = signal location?
  • r =rail direction/path (two or three letters related to line, underscore, 1 or 2 with appended letter for lines with branches)
  • vi = ID
  • lng = longitude
  • l = trip text identifier
  • lat = latitude

Interesting to note is that this service appears to offer higher resolution longitude and latitude data. It’s funny that Tripview also includes a heading which really strictly doesn’t seem necessary. They also seem to have set everything up ready for when live data is available on all lines which is very smart!

Through some playing around, a multiple-line query can be crafted by appending lines with commas – e.g. http://api.transittimesapp.com/rt/sydney/vp?c=BLU,WES which provides the data for Blue Mountains followed by the Western line. This technique might be used in the map view of train movements which is not available in the free version which only supports a line-by-line live track by deleting previous favourite and adding a new live track favourite.

Really, in order to make proper sense of the fields, one has to either guess, be the developer, or have access to the file which describes the trips, stopping patterns, shapes etc. Unfortunately, unlike the bus situation where the data was being offered openly (static data, not live data), the train data doesn’t seem to be easy to get. Kudos to Mr. Zervaas – and hurrah for another source of data for those who might be interested, although again, none of this should be used commercially or on a continuous basis as it represents an unfair burden on these app developers to fund the server resources to provide the data. I consider this an “added bonus” extra for those who might have the inclination to look at these things as a bit of a “mental exercise”.

Conclusion

I think it’s clear why Grofsoft’s Tripview is my favourite. It’s easy to use, straightforward, does what you need to do and is efficient at doing it. Having experienced these alternatives, even the user interface leaves me rifling through menus galore – it’s just not what you want when the train or bus is within earshot. The other advantage is its simple protocol which is easy to dissect – being one of the only ones (aside from TransitTimes+ which as I have discovered, is also amenable as a data source) I can see which the more adventurous can “leverage” the data for – responsibly of course.

About lui_gough

I'm a bit of a nut for electronics, computing, photography, radio, satellite and other technical hobbies. Click for more about me!
This entry was posted in Computing, Travel and tagged . Bookmark the permalink.

2 Responses to Checking it out: Cityrail Real-Time Train Information

  1. Pingback: Visualizing Recorded Cityrail Live Train Data on Google Earth | Gough's Tech Zone

  2. JD Davison says:

    Hi Gough, This article is very relevant to what I was searching for, although I was really hoping that one of the apps would connect directly to a centralised API such as RMS, Sydney Trains, etc.

    I was looking for the data that you’ve posted above, but it seems some is missing. I’m happy to do a MitM attach if you need (I do them often for reverse engineering).
    I’ve added you on Google+, let me know if you’d like to see the output of the others.

Error: Comment is Missing!