SLTF Consulting
Technology with Business Sense



The trick to hot-plugging is not frying the module-or your patience

Scott Rosenthal
January, 1995

Next to the edict against removing tags from mattresses, the most ingrained rule of life in the 20th century is that anytime you want to do anything to an electronic device-be it a computer, TV, stereo or printer-you must first turn off its power. The point of this rule is twofold. While it serves to protect us, it also protects the device. Most electronic systems hate to have people removing or installing parts with power on and express that displeasure through things like sparks, fire, smoke and assorted acrid smells. However, sometimes-and I've only personally encountered this requirement once in my career-you must design a system that allows someone to insert or extract components while the power is on. Even though I'm not an expert when it comes to this strange world of "hot-plugging" devices, I've learned a few tips that should allow you to skip my initial mistakes and instead develop something that lets you sleep easily.

Pod people

My sorry tale begins with a simple customer requirement: design a removable pod that allows a user to move a sensor and its associated calibrated circuitry between instruments without turning them off. Full of confidence (and naiveté) I said sure, I could provide that capability. However, hot-plugging isn't simple, and the list of problems I encountered designing, implementing, and "productizing" this interface is too long and unpleasant to cover in minute detail. Instead, I'll list some highlights and explain how you can avoid similar glitches.

The first problem I ran into was the physical interface itself. For some reason the customer wanted to use a 3" long connector with three rows of pins. Even though the pod floats in its slot and seats on two guide pins, I bent more pins trying to insert this pod then I ever did plugging in an EPROM. The complication is that the pod must slide down a long chamber and mate with male pins at the bottom.

Another problem with the connector, especially at the beginning of the project, was that the pod chamber's mechanical design resulted in the pod approaching the fixed connector at an angle. This misalignment caused the lower row of pins to occasionally make only intermittent contact. To prevent both problems, the ideal solution is to minimize the number of signals on the connector so you can use as few pins as possible. In addition, those pins should occupy as small an area as possible. In one HP instrument I once saw a beautiful round connector specifically meant for hot-plugging.

Finally, in specifying the connector, the customer made all the pins the same length. This fundamental design flaw precluded the computer from detecting when the user had firmly seated the pod, or when he started removing the pod. Talk about asynchronous events-one minute the computer's happily using the pod when suddenly, without warning, it's gone! With some short pins in the connector carrying interlock signals, the processor could verify that the pod is all the way in by making sure those pins present the interlock signal. Likewise, if the system operator pulls the pod out, the short pins break contact first giving the host CPU a few milliseconds warning that the pod is going away.

Powering up

Assuming you've gotten the pod into its slot and all the pins are mating correctly, now what? The first thing the computer must do is detect that the pod is present. Simple enough, just install a pull-up resistor on the motherboard signal that also goes through the connector to a grounded pin in the pod. When the user inserts the pod, the line pulls Low and the computer merrily goes ahead and starts using the pod. Not!

As with all mechanical connections to a computer, the first step is to debounce the signal, but a more significant problem lurks. Just because one pod-detect pin somewhere on the connector is making contact, you can't assume that all the other 60 zillion pins are making contact. A better situation is to assign interlock pins at each corner of the connector as well as one in the middle (remember, connectors can flex). Unfortunately, I didn't have the luxury of dedicating pins to this purpose and thus had to identify other signals on the connector that could also serve this purpose. I came up with some rather messy techniques that seem to work, but didn't offer peace of mind.

Dead connector

When it came to actually controlling power to a hot-plugged pod, I initially installed MOSFETs on all power lines as well as buffers on all computer signals. With the pod out, the computer sets all its signals Low, turning off the MOSFETs and buffers. I assumed that plugging a pod in wouldn't be a problem because the connector was dead. Unfortunately, problems began appearing almost immediately. Inside the pod was a real-time clock (RTC) chip for keeping track of elapsed time. When the pod was out of the instrument, the power for this chip came from a lithium battery. Occasionally, plugging the pod into a seemingly dead connector caused the RTC to loose its time information. I traced this problem to a floating MOSFET output from the instrument that had a small charge on it. This charge dissipated through the RTC before the computer powered up the pod, thereby wiping out RTC contents.

I also had trouble with the computer's power supply. When it tried to power up the pod, the computer reset! The problem was that although the pod normally draws only approximately 100 mA, on power up it exhibited an inrush current 100x larger than its steady-state level. This surge created a disturbance on the computer's Vcc line that caused the power-supervisory chip to reset the computer. The solution I chose was for the computer to pulse the Power Enable line to the MOSFETs, thereby building up a charge in the pod without overtaxing the instrument's internal power distribution. This technique worked, but again it was a patch.

Which way did it go

One of the biggest problems I faced with the removable pod was the fact that a user could yank it out while the computer was trying to write information into a serial EEPROM inside it. Even something as lowly as a floppy-disk drive has an indicator light to indicate disk activity. People quickly learn that if the light's on, they don't touch the disk. Unfortunately, the pod didn't have an activity light, and the original specification never called for asynchronous writing to the serial EEPROM. In the end I had to implement a complicated memory scheme that allows the computer to reconstruct the pod's information if a user happens to yank it out during a data write. The lesson here is don't implement hot-plugging without some mechanical interlock that allows a user to remove the component only when the system is ready.

This effort was necessary because even though you might calculate that the odds of having a problem like this are one in a thousand, with 100 instruments in the field and operators installing/removing pods ten times a day, you'll encounter problems daily.

The final option

While considering all these design features, keep one more aspect in mind. Hot-plugging isn't easy, and it's far from foolproof. Given all the possible error modes inherent with a hot-plugging device, you must decide if the risk is worth the benefit it provides. If the worst possible failure is acceptable, then hot-plugging might be an acceptable solution.

Putting this argument in perspective, consider that my PCMCIA standards book says not to hot-plug memory cards because the interface design can't guarantee the memory contents. This statement comes from a standard that everyone believes allows hot-plugging of cards! PE&IN

Copyright © 1998-2012 SLTF Consulting, a division of SLTF Marine LLC. All rights reserved.