PACIFIC

MOUNTAIN

CENTRAL

EASTERN

 

Archive for the ‘Tech Tales’ Category

Tech Tales: The case of the bad protocol

Thursday, June 24th, 2010

Editor’s note: ATM Network technicians have the experience to solve even the thorniest problems, and routinely go above and beyond to do so. This is one such story.

One day the ATM Network service department got a call from a bar and grill in southern Minnesota. Their ATM had suddenly stopped contacting the transaction processor, rendering it useless. When it printed receipts, they said “System unavailable.”

The technician had the owner print out the machine’s electronic journal, which showed that the the ATM was running into “protocol errors”. That usually meant that transactions were getting interrupted in the middle of processing. The most common causes all involve the phone line: too much static, interference from a DSL connection, a shared phone line or (for technical reasons), phone service provided by cable companies.

Further questioning, however, revealed that the bar didn’t have cable TV, much less cable phone service. It didn’t have an Internet connection of any sort, so there wouldn’t be any DSL interference. And the ATM had its own dedicated phone line.

That left static on the line. The tech called the local phone company, which checked its lines and said they were fine. But just in case, they installed a DSL filter to block DSL interference.

A couple of days passed, and the customer called back: the ATM still wasn’t working. In the meantime, the techs had gotten another call from a customer in a neighboring town. He had two machines: One was on an Internet connection, and it was working fine. The other used a phone line, and it was having exactly the same problem as the first customer.

The tech asked which phone company owned the line. It was the same company that served the first customer. This wasn’t unusual: the company serves a large swath of southern Minnesota. The tech called the company and told them a second machine was down. The company checked that line, too: it was fine.

Then a third customer called with the same problem. Different machine, different model – but the same phone company.

The tech thought about it for a little bit, then looked up the phone company’s service area and began calling ATM Network customers in the area. He found four more clients with ATMs that couldn’t communicate with the transaction processor.

He called the phone company for the third time and told them what he found. They still insisted it wasn’t their fault, and suggested it might be ATM Network’s server.

The tech seriously doubted that, but to be sure he called up merchants who had ATMs from competitors that didn’t use our processing network. They, too, reported processing problems.

He called the phone company a fourth time. The company said it couldn’t be their fault, but they’d look into it.

Two days later, everything started working again. The phone company never admitted anything.

Tech Tales: The Case of the Silent ATM

Wednesday, January 6th, 2010

Editor’s note: ATM Network technicians have the experience to solve even the thorniest problems, and routinely go above and beyond to do so. This is one such story.

About a month after installing an ATM in a bar in western Minnesota, the machine stopped communicating with the transaction server, making transactions impossible. A tech went down to check it out.

The machine worked properly when it was installed, so the technician figured that a component must have gone bad. He replaced the communications cable and several circuit boards, including the mainboard, but it still didn’t work. So he called in our troubleshooting team.

The first thing the team did was re-replace everything, just to be sure. Nothing helped.

Then they examined connections. The bar had a local computer network. The machine was hooked into that, using the network’s Internet connection to transmit transactions instead of a dedicated phone line.

So they began to ask the bar owner questions: Had there been any additions or changes to the network? Any differences at all? No. Further, the owner’s computer used the network for Internet access, and it worked just fine. That suggested the problem was with the ATM, not the network.
Then the bar owner mentioned that he had installed some security cameras in the bar, and that they weren’t working, either. And come to think of it, the ATM had stopped working after the cameras were installed.

The team went to look at the cameras. They were hooked into the network so that they could send their images to the owner’s computer. But why weren’t they working? And how could they have affected the ATM but left the owner’s computer unharmed?

On the Internet, central computers called DNS (domain-name servers) keep track of the location – known as “IP addresses” — of everything connected to it. Think of them as a giant directory, listing the address of computers so that messages can be sent and received from them. If a machine isn’t listed in the DNS, it’s invisible.

Networks like the one installed in the bar connect to the Internet through a router. Each router does two things: it keeps a list of devices connected to the network, and serves as the gateway connecting those devices to the Internet. That way the DNS only needs to know the address of the router; the router takes care of distributing incoming and outgoing messages for everything on the local network.

There are two ways for a router to handle addresses for its devices. A device can have a fixed, “static” address that never changes, or the router can assign a new “dynamic” address every time a device connects to it. Dynamic assignment is generally preferable, because it doesn’t require manually setting addresses for each individual device. It’s managed using a DNS setting on the router.

A quick check showed that the cameras and the owner’s computer had static addresses, while the ATM used a dynamic address. Which was confusing, because of the two devices with static addresses, one worked and one didn’t. The problem wasn’t as simple as “static good, dynamic bad.”

Thinking it over, the team decided that maybe the problem was with the router. So they checked it – and discovered that the DNS settings weren’t enabled for dynamic addresses. As far as the ATM could tell, it was properly connected, but its communications were hitting a wall at the router.
The owner enabled the settings, and the ATM worked perfectly.

But how had the problem occurred in the first place? Remember that the ATM worked when it was first installed, then stopped working when the cameras were installed. And the cameras, despite being on static IP addresses, still didn’t work.

After another round of questions, it turned out that the camera installer inadvertently disabled the router’s DNS settings during installation, knocking the ATM off the network.

And the cameras themselves? They weren’t working because of a separate installation error. Their problem was totally unconnected to the problem that took down the ATM. But it *looked* related, complicating the diagnosis of the problem.

Tech Tales: The Case of the Tampered Keypad

Monday, November 9th, 2009

keypad

Editor’s note: ATM Network technicians are so experienced that they routinely uncover and solve problems that the manufacturers themselves missed. This is the story of one such incident.

Because ATM transactions involve money and personal financial information, they are heavily encrypted – the data encoded so that neither the ATM owner nor anyone who sees the data stream can read it.

Specifically, whenever a customer types information — like, say, their PIN number — into an ATM keypad, the keypad takes the number and runs it through a mathematical algorithm that encodes the PIN so that only the transaction-processing server can read it.

Because the algorithms are the key to reading the encrypted information, they’re treated with a level of security normally reserved for nuclear launch codes. The manufacturer keeps control of the algorithm, and distributes a randomly generated “master key” that can decode it. The key is broken into two halves. The manufacturer keeps one half; the other half is split in half again and the pieces sent to ATM Network. Each piece is entered by different people, and the ATM downloads the manufacturer’s half from a secure server. So no one has the whole key.

A few years ago, a major manufacturer’s machines began dropping their encryption. Rather than send unencrypted information, the machines took themselves out of service, displaying an error code saying the keypads had been tampered with.

Except the keypads hadn’t been tampered with. Plus it was happening a lot: some merchants were calling twice a week. And every time it happened, the only way to fix it was to install a new master key — which meant sending out two techs with the two halves of the code, or sending out a tech with one half and mailing the other half to the machine’s owner. Either way, it was a lot of expensive service calls.

Our troubleshooting team got involved. Since the error code mentioned keypads and the keypads were part of the encryption process, they began replacing keypads and sending the old keypads back to the manufacturer for examination. That sometimes solved the problem, but more often than not the replacement keypads would fail, too.

The manufacturer wasn’t being particularly helpful, so the team sat down to examine the keypads.

Each pad had a circuit board on the back that contained the encryption chips. The board drew power from the ATM, with a battery backup in case the power went out. The team tested the board, checking connections, looking for broken circuits and so on. Everything seemed fine.

Then a tech noticed that the backup battery appeared to be loose. Closer examination revealed that it wasn’t soldered to the board. A quick test confirmed that it was able to move just enough to disrupt the circuit bringing power to the encryption chips.

But so what? It was the battery backup, not the main power line. The problem would only affect the machine if it lost power.

Unfortunately, the machine was designed so that resetting any error code — including “out of paper” or “out of cash” messages — required turning off the power. Some quick checking confirmed that the problem only cropped up when the machine lost power or was turned off.

The solution? Resolder the battery to the board.

ATM Network alerted the manufacturer, which addressed the problem in a technical bulletin. But instead of fixing the problem by properly soldering the boards during production, the manufacturer just updated the machine’s software so error codes could be reset without turning the machine off.

Because of that, the problem still crops up from time to time. Every time it does, ATM Network technicians fix the problem permanently by resoldering the boards.

Neither the manufacturer nor any other company had managed to solve the mystery. Indeed, the manufacturer was spending a small fortune on replacement keypads. Only ATM Network had the expertise and dedication to find a solution.

Postscript: Issues like this are one reason ATM manufacturers are starting to build in remote key capabilities into their software: it allows master keys to be sent and managed electronically, increasing security while cutting down on the need for technician visits. Listen to this remote key webinar if you’re interested in learning more about them.

Tech Tales: The Case of the Missing Internet

Monday, October 26th, 2009

404

Editor’s note: ATM Network technicians have the experience to solve even the thorniest problems, and routinely go above and beyond to do so. This is one such story.

When ATM Network installed an ATM in a county licensing bureau, they set it up to use an Internet connection instead of a phone line. That was both faster and cheaper for the client, since the government office where it was located already had a high-speed Internet connection, and using it meant the ATM didn’t need a separate dedicated phone line.

There was only one problem: the Internet connection didn’t work. The machine kept reporting that it would connect to the transaction server, only to have the transaction server drop the connection in the middle of transactions.

ATM Network spent two weeks and five visits troubleshooting. Our techs met with the county’s IT department, who assured us that their network was working fine, and the problem had to be the ATM. But our troubleshooting team couldn’t find anything wrong with the ATM.

The team ran Internet traces and got a puzzling result: the machine reported that the server was dropping connections, while the server reported that the machine was dropping connections. It should have been one or the other, not both.

Finally, the team hauled the machine back to the ATM Network warehouse, set it up there, and ran a test transaction. The Internet connection worked flawlessly.

Certain that the problem was with the network, the team went back to the county’s IT department and began asking questions, tracing the exact path that the ATM data followed through the network.

It turned out that the data first went through a city router (the licensing bureau was located in a city-owned building), then a county router, then a state router before being sent on to the transaction server.

The team tracked down each router and examined them. They discovered that the middle router — the one owned by the county — had a web filter on it.

A web filter is software that restricts access to certain sites. So if you don’t want your employees playing online games, you would set your filter to block the addresses of known gaming sites.

For some reason that filter had decided it didn’t like the address of the transaction server, and was blocking it.

Rather than totally disable the web filter or risk it blocking transactions again later on, the team gave the ATM a unique “static” address and then exempted that address from the filter. The machine has worked perfectly ever since.

Tech Tales: The Case of the Missing Surcharge

Monday, October 12th, 2009

Editor’s note: ATM Network technicians are so experienced that they routinely uncover and solve problems that the manufacturers themselves missed. This is the story of one such incident.

A few years back, a major ATM manufacturer introduced a machine that offered lots of great features — big screen, modern operating system, reliable mechanisms and lots of expandibility. But its software lagged behind, making the machines run sluggishly.

So with great fanfare, the company released a software upgrade that turned the laggard into a blazing-fast wonder. Customers clamored for them, and ATM Network began installing them by the truckload.

But we quickly encountered a problem. Cash withdrawals worked fine, but every time a customer performed a balance inquiry, the machine would shut itself down for 15-18 seconds, disrupting transactions.

Our troubleshooting team got involved. After some testing, they discovered that during those 15-18 seconds the ATM was resetting itself by downloading fresh surcharge information and encryption keys from our transaction processing server. That was a normal function, but one that rarely occurred — it was only necessary when there was a discrepancy between the ATM’s settings and the server settings. In this case, however, the settings weren’t changing on either end, so there was no reason for the resets.

The manufacturer didn’t know what the problem was. So our team began asking questions. Were there programming errors in the software code? Had the software upgrade made any changes to how settings were stored? Were there new settings that the transaction server wasn’t reading correctly? The answers came back: no, no, no.

The team finally decided to focus on the odd fact that the problem only cropped up during balance inquiries. From the ATM’s perspective, the main difference between a balance inquiry and a cash withdrawal is the surcharge: balance inquiries are free, withdrawals generally are not. Maybe there was a problem with how the machine decided which transactions had a surcharge and which ones didn’t.

Programmers use something called a “flag” to determine status settings. A transaction that has a surcharge would have its surcharge flag set to “on”, meaning a surcharge is assessed. A balance inquiry has its surcharge flag set to “off”, meaning it’s free.

But as the team dug through the system, they discovered that this machine did things a little differently: the surcharge flag was always set to “true”; all it did for free transactions was set the surcharge amount to $0.

That was odd, but seemed harmless. The effect was still the same: withdrawals had a surcharge, balance inquiries didn’t.

The team did some more digging, with engineers from the manufacturer on the phone to help. Together they soon discovered the problem. Besides checking its own settings, the ATM also verified its settings with the transaction server. But the transaction server had only one surcharge amount listed — the surcharge levied for cash withdrawals. This made sense, because if the ATM used the surcharge flag properly, then the only time a surcharge would be triggered was during a cash withdrawal.

But in this case, the ATM was triggering a surcharge on balance inquiries, too. And that $0 surcharge didn’t match the surcharge information on the server. So the ATM assumed its settings needed updating, and took itself out of service while it downloaded new settings from the server.

Thanks to the combined efforts of ATM Network and the manufacturer, the manufacturer wrote and released a software patch fixing the problem.

Tech Tales: Jim’s Corner

Monday, September 28th, 2009

The dotted line shows the route driven by our tech.

The dotted line shows the route driven by our tech.

Early one spring, ATM Network got a call to install an ATM at a resort on an island in Lake of the Woods, on the Minnesota/Canada border. As you can see from the map, Lake of the Woods is huge and the border cuts back and forth through it several times, but generally the east and north sides are in Canada, while the United States owns a portion of the west and south sides.

The weird thing is that a large chunk of the western shore is a peninsula, known as the Northwest Angle. The peninsula itself is part of the United States (it happens to be the northernmost point in the United States, aside from Alaska), but the shore it’s attached to is Canadian. This odd situation is a result of a geographic mistake written into the Treaty of Paris, which ended the Revolutionary War and defined the boundary between Canada and the United States.

The resort was on an island just off the end of this peninsula, and much closer to the north shore than the south. That meant the installer had to cross into Canada, drive up the west side of the lake, cross back into the United States, drive to the end of the peninsula, and then get on a boat to reach the resort.

There was flooding all over northern Minnesota, and most of the hotels in the area were still closed for the season. Those that were open were booked up. So the installer spent the night in Grand Forks, N.D.

The next morning he tried to drive to Warroad to cross into Canada. But flooding had closed the roads around Roseau, so he backtracked and ended up crossing the border at a town called Pinecreek. After going through customs (in a van filled with tools and a large ATM), he spent 45 minutes driving toward the peninsula.

As he neared the peninsula’s border, he started seeing signs for a place called Jim’s Corner. They all urged drivers to stop there, saying “You must stop at Jim’s Corner.” “Be sure to stop at Jim’s Corner.” It sounded like the Wall Drug of northern Minnesota. By the fifth or sixth sign, the installer was really curious what Jim’s Corner was.

But when he finally reached it, “Jim’s Corner” was a plain, unremarkable shack on the side of the road. The installer shrugged and drove on past.

The shack at Jim's Corner.

The shack at Jim's Corner.

He crossed back into the United States (there was no border crossing, just a sign) and drove to the end of the peninsula, where a water taxi service could take him to the resort. He wasn’t sure what tools he would need, so he unloaded his entire van (including the ATM) into the boat.

The boat made several other stops, so it was a couple of hours before the installer arrived at the resort.

The installation itself was quick and routine. After making sure everything was working and showing the resort staff how to use their new machine, the installer packed his tools back up and went to wait for the boat to return. A couple of hours later he was back on shore. He packed up his van and headed back toward Canada.

As he drove down the peninsula, he again started seeing signs for Jim’s Corner. It had been a long day, and he was still mystified at what Jim’s Corner had to offer that justified all the signs. So when the shack came into sight, he pulled over, stopped, and walked in.

Inside was a video phone with U.S. and Canadian flags, underneath the words “Where are you checking into?” Since he was heading into Canada, he pushed the “Canada” button. A customs agent appeared on the screen and began asking him standard customs questions: “Where are you going? Do you have anything to declare?”

Yep, it was the border station, run on the honor system and managed over remote video by customs agents based elsewhere.

After answering the questions — and hoping the agent wouldn’t notice that, according to their records, he had entered Canada twice that day without ever leaving it — the tech returned to Grand Forks for the night, ending a 14-hour day.

More on Jim’s Corner
In 2006, CNN’s Gary Tuchman visited the shack.
Photos of the inside of the shack.

Tranax ATMs Triton ATMs Hyosung ATMs WRG ATMs ATM Network signs Wireless ATM adapter ATM security Check collection ATM paint or ATM wrapping ATM parts Custom ATM cabinets Buy ATM paper