Debugging in -25C...

Howdy!

The name of this post kind of says it all. Sitting in the car with a laptop and tapping on the keyboard while exhaling visible moisture which might make me look like a living steam engine. Trying to fix relentlessly the few remaining bugs, hoping that the laptop battery would last long enough, even though part of me wishes that the laptop would generate more heat.  At least the parking heater works!

For the past few days I've been building the remote heater starter and finally ChiliCAN v0.1 was born! It resembles little bit those implementations that I wrote about in the first post, but don't let the version number confuse you. This one speaks CAN and has multiple additional functions in addition to it's raison d'être: flipping on the relay and making the car a warm haven amidst the hostile environment that the Northern Finland is now at this time of the year.

Let's see what the little controller box has eaten: 

Ok,  it's not going to win any beauty contests, but it does what it's supposed to do.

The schematic is quite similar to what Arduino and the CAN shield would be when combined together. On the left is a separate USB breakout board, since it is not needed unless there is a need for debugging, giving manual test commands via the COM port or re-flashing the firmware. So the list of components is quite familiar: Atmega328p, MCP2515 CAN controller, MCP2551 can transceiver. A relay for grounding the AEM control pin (gives better isolation than grounding it directly via a transistor). The 7805 constant +5V voltage regulator + few passive components. Under the ribbon cable is also a SPI header for easy access to debugging with oscilloscope or maybe BusPirate (actually I ordered it a while back, but haven't yet tested it).

There's a extra 5A car fuse, isolated and kept in place with a hot glue, but now it's kind of deprecated since I have a main fuse outside the case for extra safety. Also the connectors are insulated with hot glue, and that with the IP65 protected case should give ample protection against moisture.  I still have to spray the board with a coating of protective lacquer, but that's only after extensive testing.

Then there's the female headers, resembling little bit the headers in Arduino board. And what's with the antenna?

Yup! That's the Arduino GPRS shield based on the SIM900 chip. I didn't have guts to start dismantling it, so I installed the Arduino type headers and laid the shield on top of the main board. I didn't want to use the built-in SMA-connector on the right side of the shield since that would cause both placement problems as well as increase my worries of poor insulation. So I used a left over WiFi SMA->U.FL extension wire from the Asus Maximus V Formula motheboard that I recently installed. I didn't have any use for desktop WiFi, so why not put the nice cable to use. Actually quite many of the parts here are left-overs or parts that have been recycled from old devices. There was also a long WiFi antenna included with the mother board, but I decided to use the one shipped with the GPRS shield.  

Sadly after preliminary testing I had to conclude that the Arduino+GRPS shield+CAN shield combination isn't going to work so nicely for two reasons. Separately they work quite fine, but all the combined code requires more memory than the measly 2Kb that theAtmega328p has to offer. This resulted in numerous random crashes, weird behaviour or just lack of functioning. This problem could be solved with optimization tricks, such as better memory utilization by removing unnecessarily large parameters and variables, and moving some parts out of SRAM to flash memory etc. 

However the much bigger problem would emerge when looking at the power consumption. Even when the sleep mode was enabled (in which text messages and calls could still be received) , the total amount of juice sucked by the device was around 80-90 mA. Although the sleep functionality of Atmega and CAN controller are not yet utilized by my code, the portion of power consumption by the SIM900 is almost half of the total even though it is in sleep mode ! Maybe I'm doing something wrong here, since according to the specs the GPRS module should consume only around 1-2 mA when dozing. 

For now as a heater start signal I will be using the yellow light button on the wireless key fob. Luckily the parking spot for my Volvo is almost right below my apartment window, so it is in the wireless range of the key fob and I will be able to use it to start the heater from the warmth of my house :) 

Pressing the light button 4 times (turning the car interior and exterior lights on-off-on-off in a time period of 10 seconds will start the heater. Turning it off is done in the same manner. I can also check the heater status with the fob: When pressing the light button  (regardless whether the sequence ends up in starting to heater or not), the car headlights are blinked 1-4 times. One blink means heater is off, two blinks: heater is either starting or stopping, three blinks: heater is on, four blinks: there is an error condition.

Here's the device ready for testing! Leftmost green led is for status. Next (yellow) is CAN traffic indicator (actually it doesn't blink for every CAN frame, since then it would just stay lit when connected to the car because of the huge amount of CAN traffic going on all the time. Instead, it blinks only when sending messages or receiving ones that have some inherent importance to us.).The third LED is a heater status indicator and fourth one is an error LED. I might later print labels and other stuff on the case with silk screen technique.

Here is the wiring harness for extending the AEM connector, nicely sleeved for making it look less like a botch job for what it really is. I tried for quite many hours to find suitable male/female connectors compatible with the 4-pin one that AEM and its harness has, but without any result. So I ended up having 4 separate connectors, one for each AEM pin. 

Then there are two DB9 standard CAN connectors. Leftmost has 12V connected to its pin #9 and is meant for connecting the ChiliCAN, and the right one is an extra debugging/tracing/logging CAN cable carrying only CAN-L, CAN-H and ground wires. Then there's extra fuse housing and the screw terminal to put the wires together. That + numerous cable ties give pretty good strain relief. 

I have to remind you that this is still going to be temporary installation as I may not even keep the AEM beyond the first few days of testing, so this not-so-elaborate setup will have to do for now.  Ok, so maybe I'm going to keep these in place until when outside temperatures are easier to count in celcius degrees than Kelvins.

Here's the final setup. Installing these separate dangling wires to AEM connector with  "relative safety" requires either that the negative wire from the car battery is detached or at least that the AEM fuse is removed. Also good electrical insulation tape is necessary. Did I mention botch job?

Later I might bypass the AEM connector altogether and connect the ChiliCAN directly to the Rear Electronic Module (REM) where the AEM wiring is originated.

Inspired by the stuff the guys at the Swedish Volvo forum have been doing, I too decided to add menu feature to the ChiliCAN. The menu can be accessed by keeping the cruise zero button pressed for two seconds.  The +/- buttons are used in navigating and another zero button press makes a selection. The return button exits the menu.

Using cruise buttons doesn't interfere with the normal cruise functionality unless we have turned it on before entering the menu. And even then the cruise buttons work normally and are not hijacked in any way.  Another, more familiar way of using the DIM menu would have been with SWM control stalk, but then we would have been forced to hijack the control stalk and I'm not sure if that can be done via CAN bus.

Heater can be turned on and off via the menu. Here it's status is shown (off / on / starting / stopping / error). It's handy to see the battery voltage without having to go crawl under the hood (or in case of S80, the trunk) with a multi meter. The various bits of information are queried by the ChiliCAN from CEM via diagnostic messages.

 It can be quite informative to see the coolant temperature here, since it gives rough estimation of how cold the engine is and whether you should start the heater. This makes it much easier to do rough guesses of how long should you keep the heater on to get the cabin and the engine comfortable. And of course, this isn't limited to the diesel/petrol parking heater, but could be used in conjunction with any additional heaters. When my Ardic broke down a while back, I got an electrical block heater (coolant heater) installed, but I wasn't (and still I'm not) quite sure how effective it is in hard metrics. With ChiliCAN I can now easily be able to see how well (or badly)  it really works.

One feature to tinker in the future would be the enabling of the water pump in Ardic in conjuction with the electrical coolant heater. That would give more even spread of the heat, since the block heater doesn't have its own water pump and is just heating one part of the coolant circulation, hoping that the difference in thecoolant temperature itself moves the liquid around a bit.

 Here we can see the GSM module status. Code isn't included in the firmware yet, since the memory optimization hasn't been done yet.


There is still quite a lot of things to do, but priority is now in lowering the energy consumption. After that, I might start thinking about having features little bit like what Volvo has to offer in their newest models: Volvo On Call mobile app :)

FFffuuuuuuu......

Succession of failures. What an oxymoron.

A week ago I continued testing the ignition of parking heater with diagnostic commands. Last time I checked, it worked without the key in ignition using VIDA, so I took another shot with ELM327. No go. Ok, started injecting the periodic keep-alive message to CAN bus and tried again, but without any success. Then I put the key into position I, and what do you know, the heater started. It DOES need the key in the ignition after all, so there goes my plan to implement the remote control using the diagnostic commands. <insert swear words and cursing here/> My previous success with VIDA must have been because of some kind of inherent timeout in the CEM. Diagnostic codes and commands work for some period of time after the key is removed from the ignition, but after a while (maybe because of security reasons) the CEM stops responding to them.

There is a way to spoof the other modules into thinking that the key is in the ignition by overwriting the continuous status updates made by CEM about the key position, but there's no workaround when it comes to CEM itself. It would be like breaking into you own car. So I folded in front of this another unforeseen obstacle, took the path of less resistance and ordered the AEM module. It's going to cost little bit over 200 euros (including software updates to CEM and AEM itself), sure, but I didn't see any way beyond the key problem and I'd spent enough time tackling this one.

BTW. Guys in King of Thrones are wrong. The winter isn't coming. It's freaking already here! Few days ago the temperature went down to -22 celsius ( -7 fahrenheit)!  And that resulted in the next system failure...

The parking heater stopped working. The day before this total failure it already gave a hint for the upcoming problem. For some reason it hadn't reacted to previously set timer, but did go off when starting it manually.  Next day, I had set the timer again, but when I approached the car, I could see that everything was still frozen. Attempted starting the heater manually, and it did run for a minute or two, but then stopped again. Tried it for the third time, but the same thing happened.

Now after three unsuccessful start attempts the CPM goes into lock-down that can only be reset with VIDA or other expensive diagnostic device. Poor man's choice, Torque + ELM327 won't do, since they won't be able to query CPMs fault codes and reset them. So, my efforts developing the Sardine CAN and playing around with VIDA weren't total waste of time. Lucky me, except that I was visiting my parents' house and I didn't have the device with me, so I had to resort to firing up the engine in -22C since there isn't any additional heater installed in the car.

Except now the car would start. Motor would cough for a second or two, but then the start motor just continued to spin. Now I was starting to get pissed off.  This sounded more like a battery problem or frozen fuel lines, since a problem in either one of those could affect both the motor and the heater. There was no low voltage light on the DIM lit,  but I didn't have my multimeter to check the battery voltage. The artic diesel variety that is offered here in Northern Finland is supposed to handle at least -29C temperatures without gelling/waxing and clogging the fuel filter and the fuel lines. Anyway, I didn't have any way to pinpoint the problem at this stage. Luckily I managed to start the engine after covering it with blankets and letting two powerful heat blowers warm it up from below a bit. Even that took over 2 hours.

So I got home, connected my dear Sardine CAN, fired up VIDA and started snooping around. First, the fault codes:

Now this is interesting. There's no way simple fuel clogging problem would result fault codes piling up as they do in this list. Especially CEM-6C48 is meaningful in this context. It would indicate there was a communication problem between CEM and the transponder implanted in the ignition key. It could be caused by many different reasons (two different keys next to each other in the keychain, electrical interference, wrong key etc), but end result would be the same: Fuel delivery is inhibited to fuel injectors and motor wouldn't start. However this wouldn't have any effect on the heater, since its fuel input is controlled by heaters own fuel pump which is independent from any security measures related to engine. Anyways, fault codes this many in so many different modules would be unlikely to occur for reasons indicated in each fault code. There must have been some kind of electrical connectivity problem because of the weather, most likely in the CEM or one of its harnesses. If this ever happens again, I will have to dig into this, but for now, I reset the fault codes and wait if any of them should pop up again.

Now that the fault code for heater (CEM-5F4F : Too many unsuccessful start attempts) was also been cleared, I was free to try starting up the heater, but with this time I had VIDA to guide me with troubleshooting.

Glow plug, combustion fan as well as water and fuel pumps were working normally, but the flame sensor read constant voltage. According to VIDA, voltages from 2.5 to 5.0 are interpreted as "no flame" and voltages below 2.5 volts mean there is a flame in the heater. The CPM waits for a while for the diesel to ignite, but after 1-2 mins it gives up, stops the heater and increments the error counter. So, either the flame sensor has broken down or is covered up, or the diesel fuel doesn't reach the heater. Anyway, the problem most likely isn't going to go away magically and repair shops usually ask between 400-600 euros + parts for the repair, so I think you're going guess where I'm going with this..

Next: Ardic tear-down!

Success!

It's Saturday night and I feel like partying! :D  After countless hours and ~3500 lines of code later, I finally managed to connect VIDA successfully to Volvo, launch the diagnostic part related to Combustion Preheater Module and turn on the parking heater with my laptop!

Sorry about the poor image quality here:

Vida correctly identifies most of the vehicle features. Only transmission, steering and body style had to be manually entered. Reason for this can be seen in the next picture..

Sardine CAN is connected to low speed network, so all queries relating to CAN modules residing in high-speed network (such as Break Control Module,  Staareing Angle Sensor, Engine Control Module, Transmission Control Module as well as the high speed interface of Central Electronic Module) cannot be reached. For some reason messages from few low speed modules ( Upper Electronic Module, SRS, Rear Electronic Module) are not received correctly either. Accessory Electronic Module and Road Traffic Information module I don't have in my car.

When sniffing the CAN traffic while VIDA scans the modules and their diagnostic error codes, I can infer which module is being queried and which CAN identifier the module uses for replying. Note that this identifier differs from the one the module uses for normal inter-module communications.

Let's refresh our memories of the general format of module query (from past blog post):

000FFFFE CB xx B9 F0 00 00 00 00
          |  |  |  |
          |  |  |  |
          |  |  |  '---- Identify (?)
          |  |  '----------------- Read Data Block By Offset
          |  '-------------------- Module id (list below)
          '----------------------- Message length
00 0F FF FE: The identifier VIDA (or any other diagnostic module) uses for messaging.
Message length: High nibble seems to be always 'C' in command message. Low nibble: Bit 3 is always on. Bits 0-2 is the actual message length (excluding the first byte). Hence A=2, B=3, C=4, D=5, E=6, F=7

I found this command set somewhere on Swedespeed car forum:

A1 No Operation Performed (keep alive)
A3 Security Access Mode 
A5 Read Current Data By Offset 
A6 Read Current Data By Identifier
A7 Read Current Data By Address 
A8 Set Data Transmission
A9 Stop Data Transmission
AA Dynamically Define Record
AB Read Freeze Frame Data By Offset
AC Read Freeze Frame
AD Read Freeze Frame By DTC
AE Read DTC
AF Clear DTC

B0 Input Output Control By Offset
B1 Input Output Control By Identifier
B2 Control Routine By Offset 
B4 Define Read Write ECU data 
B8 Write Data Block By Offset 
B9 Read Data Block By Offset 
BA Write Data Block By Address
BB Read Data Block By Address 


And here's the list of all modules that were queried and identified on Volvo S80 MY02.
CAN diag Id    ID  Description 
00 80 00 03 :: 40  CEM, Central Electronic Module 
                   (also answers queries related to CPM(heater)
00 80 00 09 :: 51  DIM, Driver Information Module
00 80 08 01 :: 48  SWM, Steering Wheel Module
00 80 10 01 :: 29  CCM, Climate Control Module
00 80 00 11 :: 43  DDM, Driver Door Module
00 80 00 81 :: 45  PDM, Passenger Door Module
00 80 01 01 :: 2e  PSM, Power Seat Module
00 80 04 01 :: 46  REM, Rear Electronic Module
00 80 02 01 :: 58  SRS, Air bag
00 80 20 01 :: 47  UEM, Upper Electronic Module
00 80 00 05 :: 60  AUM, Audio Module
00 80 00 21 :: 64  PHM, Phone Module

These module were queried but didn't reply:
ID  Description
50  CEM, Central Electronic Module (Hi-speed interface)
01  BCM, Break Control Module (hi-speed network)
52  AEM, Accessory Electronic Module 
11  ECM, Engine Control Module (hi-speed network)
28  SAS, Steering Angle Sensor (hi-speed network)
6e  TCM, Transmission Control Module (hi-speed network)
62  RTI, Road Traffic Information module

And here's the sweet sight of hard reverse engineering work coming finally to fruition! Only coolant water temp and heater work status are being correctly queried though. Few software glitches still remain, but I don't care about that for now, since the thing I've been hunting for past few weeks has been now identified! Yes, the command for starting the heater :)

Turn on the diesel heater:
00 0f ff fe | cf 40 b1 5f 3b 01 01 84 
And the reply:
00 80 00 03 | cc 40 f1 5f 3b 00 00 00 


Turn off diesel heater:
00 0f ff fe | cf 40 b1 5f 3b 01 01 80 
Reply:
00 80 00 03 | cc 40 f1 5f 3b 00 00 00 


Now, this seems weird, since I had already tried this command before and it didn't work! It is one of the possible permutations of the message I was advised to try earlier by Swedish hackers (thanks again guys!), and I'm quite sure I tried this one before. There are few possible explanations:

1) I somehow managed to screw up sending the message using ELM327 (with its yucky AT command set), but now when using the MCP2515 based Arduino CAN shield the message is constructed correctly.
2)  ECU needs something else in addition to the command message itself. When looking at the message log, I see VIDA sending the following message every 1-5 seconds:
00 0f ff fe | d8 00 00 00 00 00 00 00
Could this be some kind of keep alive message needed by ECU?

Also VIDA keeps querying following stats every 3-4 seconds and their presence could be necesssary (although unlikely):

Cmd:   00 0f ff fe | cd 40 a6 1a 04 01 00 00
Reply: 00 80 00 03 | cd 40 e6 1a 04 1e 00 00 
The 6th databyte of reply seems to coincide with ignition key lock status:
1e = ignition II, 1d=radio (ignition I), 1c=off, 18=key out

Cmd:   00 0f ff fe | cd 40 a6 1a 02 01 00 00 
Reply  00 80 00 03 | cd 40 e6 1a 02 60 00 00 
The sixth databyte of reply fluctuates between 5d and 62, and could be the battery voltage. If we assume bits 0-2 consist of fractional part and bits 3-8 the integer part, then the values here would be interpreted as 11.625 and 12.25, and would fit well in our hypothesis. Actually a battery charger was connected during testing, so voltage over 12 volts would not be strange here.

VIDA needs the key to be in ignition II position in order to launch the heater section, but I did try switching the key position and it didn't have any effect on the result itself: Heater can be turned on with diagnostic command even when key is not in the keylock! This is actually quite a relief - spoofing the keylock position in the remote heater starter would require quite a bit of more work, but luckily this doesn't seem to be needed. However what is little bit alarming, is that any indication on the heater status is NOT shown on DIM, nor does the manual on/off functionality on the control stalk work when turning on the heater using this diagnostic command. Thus I will have to put some other kind of stop functionality and warning system in place when designing the box.

Still this isn't a fully functional J2534 device yet: It doesn't support ISO9141 or any other kind of protocols apart from CAN and ISO 15765, nor does it work when connecting it to OBD port, since it's missing the K-line initialization and keep-alive messaging to keep the diagnostic relay open on CAN bus pins. Maybe I will add some more functionality later, but for now, I'm quite happy with the results that I got. Also, no more Win32 programming for a while :)