CAN Bus: Massive Noise on the bus when a power converter starts

[KMew]

Hello,

We have been using CANbus as our form of communication for a system with multiple nodes. Each node was individually tested with CAN and all of them work without issue. However, when we started system integration testing, we noticed significant noise on the CANbus channel that caused data to stop. 

To give more details, we are running 2 battery packs, a power electronics DC/AC converters (to create a 120Vac signal) and a central supervisory controller, which are all using STM32 MCUs to communicate to each other via CANbus. We are using an STM32H7AGI6, STM32F401RE, STM32F303RE. When we send a CANbus command to start the DC/AC converter, that is when the noise appears and we cannot communicate with the MCUs until the DC/AC converter is turned off.

 

We have termination resistors at the end of the bus and I have verified that the 60 Ohm resistance is seen on the bus. I've also attached a copy of our schematic for the CANbus, which all of the nodes use (except that the 120 Ohm resistor is only populated in the two connected at the ends)

I've attached a picture of the waveform too, where the two "messy" segments are when we try to start the DC/AC converter. The picture below is the CANH line, with the probe grounded to the GND pin in the CANbus node.

 

 

 

Can anyone give me some tips to improve this or prevent this from happening?

 

[MHoll.2]

Hi 

The Can-Bus is normally very resistant in noisy environments. so if you lose completely the communication you could have one of this problems:
1) check that you have a good common ground on the system (Can is differential but need a GND path for the current flow).
2) check that the cables you are using are twisted and have a impedance of about 120 Ohm
3) if You still have problems you can use an isolated CAN Transceiver or try to reduce the interference generated by you DC/AC converter.

Martin

[KMew]

Hello MHoll.2,

Thank you for your reply. We are trying to avoid major harware changes like a new CAN Transceiver (it'll be our last resort, if we have to do it). So let's start with the easier things.

 

1) When you have "have a good common ground," can you please explain what you mean by that? We have a centralized "CAN Hub," which has the three pins (CANH, CANL and GND) all connected together for all the nodes. Is that what you mean?

2) Yes, we are using twisted pair cabling for all the wires (76030 TL001 cable) and the two ending nodes have a 120 ohm termination resistor. I verified the measurement, as I see 60 Ohm between CANH and CANL.

 

[Tesla DeLorean]

>>When you have "have a good common ground," can you please explain what you mean by that?

Well bonded, like the body/frame of a CAR, so it's very difficult for the units in the system to have a different ground potential with respect to each other. Not just some 25 AWG

[LCE]

My guess (without having seen any PCBs or circuits): system problem, probably caused by the GND connections (loops!) as already mentioned, and probably noise created by the DC/AC converter.
If the system GND would be absolutely "identical" (zero impedance from node to node - which is virtually impossible), and there was some noise on system GND compared to "Earth", that would not matter, as all nodes ride on the same noise.

I would work on these things:
system design and GND connections,
the power converter and its exact flow of input and output currents,
use isolated CAN transceivers.

Concerning CAN, just for testing: have you removed the GND connections between nodes?

On the other hand, that project does not sound like there are hobbyists at work, so the power part should be generally okay...

It does not sound like an STM32 / driver / software problem at all. Which makes it more interesting. :D

[KMew]

Hello LCE,
Sorry for getting back to you so late. I wanted to try some additional tests to try and get more information. Here is some additional information:

 

- I did try to remove the GND wires from all the CANbus nodes. There was no change in results (the communication got no worse and did not get better either)

- I have confirmed that, at least, the signal I am seeing on the oscilloscope is not real. I put a CANalyzer on the bus instead of the oscillscope and, during that period that the DC/AC board is running, the CANalyzer does still see information on the bus. I do get periodic errors (stuff error, form error, etc.), but messages are still being sent. Additionally, after about 30-40 seconds of the DC/AC running, I do see that all communication stops as well. The CANalyzer stops seeing any data being sent from any node, which it is not supposed to (all nodes send their data approximately every 1 second).

- I took the setup outside of our enclosure and used a pre-twisted twisted pair cable ( https://www.digikey.ca/en/products/detail/alpha-wire/76030-TL001/12715466 ) and I do not see the communication getting dropped. So I do feel confident that there is some noise being injected and that is interfering with the CANbus. Note that this wire is not shielded, it's just a twisted pair.

- I did notice that the assembler made hand-woven twisted pair wiring, but all three wires (CANH + CANL + GND) are all twisted together, so it's not technically twisted pair

 

Does this information help at all? Because we have not been able to conclusively solve the problem

[Karl Yamashita]

---

@KMew wrote:

When we send a CANbus command to start the DC/AC converter, that is when the noise appears and we cannot communicate with the MCUs until the DC/AC converter is turned off.

---

When you say turned off, is it turned off by CAN message or manually turned off? 

I'm trying to understand when the DC/AC converter is turned off, if all of a sudden CAN communication commence?

[KMew]

Hello Karl,

The DC/AC converter is turned on and off via a CANbus message from another board, but communication is lost while it is running, so I have to manually turn it off (it's powered by a removable battery, so we just remove said battery to stop operation).

[Karl Yamashita]

---

@KMew wrote:

Hello Karl,

The DC/AC converter is turned on and off via a CANbus message from another board, but communication is lost while it is running, so I have to manually turn it off (it's powered by a removable battery, so we just remove said battery to stop operation).

---

You still didn't answer my question if after you powered off the DC/AC converter, does communication commence or do you have to power cycle the other devices to get communication working again? 

Have you tried debugging the devices that stop communication and checked if the TEC counter is greater than 127 or 255? Is ABOM enabled? You may want to check the REC count as well.

[LCE]

So better cabling already ("real" twisted pair) improved things?

I would try shielded twisted pair next.

And maybe even this:

use shielded twisted pair cable with at least 2 twisted pairs,
1 twisted pair (CAN+ & CAN-), the other pair for the GND connection, 
and with the cable shield only connected on one side, preferably with the shield not connected on the DCAC side.

[KMew]

Hello LCE,

 

We are preparing to try shielded, twisted pair right now.

 

However, we continued testing and noticed that, once we put the unit back into our enclosure, even with the "real" twisted pair, the communication continued to freeze with no improvements from when I first posted the question. So now I'm questioning the twisted pair at all.

 

I am having a very hard time isolating this problem right now.

[KMew]

So a general update of my testing. We still haven't solved the problem, but we have identified something curious. 

 

Our supervisory control board (which uses the same CAN hardware as the original post) is powered by the removable battery pack, which is the same one that powers the DC/AC. We tried a test where the supervisory control board is powered by a separate DC power supply, but kept everything else the same (DC/AC, battery pack, supervisory controller all connected via CAN, DC/AC powered by the battery, twisted pair cable, ground wire still connected) and the communication is never lost in this scenario. 

 

Does this help explain it at all? Does anyone have any ideas based on that?

[Karl Yamashita]

Did you check for reset flags? Check the power supply on the supervisor device with an oscilloscope to see if it's a consistent 3.3V with no dropout when the DC/AC converter starts.

[LCE]

Good that it's working now.

But... you are only taking care of the symptoms and don't cure the cause (EMI of the power converter).

Things like that will come back one day, probably when doing EMC tests.

[KMew]

Hello LCE,

Yes, you are right. I needed a solution to deal with the symptom, due to timelines. I have requested that, for the next revision of the DC/AC board, a deeper analysis into what is causing the conductive noise on the CANbus and a method to mitigate it has been put out for our next revision.