Target Registers Fail Nationwide

[Alpha8472]

On Saturday cash registers at Target suffered system failures nationwide. Customers waited in long lines for hours as lines snaked through the store aisles. Target employees passed out free Starbucks food and tiny Starbucks cups with drink samples.

People stared at their phones for hours while in line. Some people say it took 20 minutes to ring up 1 customer. Sounds like a regular day at Walmart.

Most people have better things to do with their time, like walking out of the store and going somewhere else.

[storewanderer]

These systems are far from fool proof. It is interesting Target did not have a back up mode or offline mode that would allow transactions to keep processing. But that causes a lot of problems as inventory tracking, pricing, etc. would not be available. Also it is strange timing for this to happen, it does not seem like a particularly busy weekend like a holiday or something. I wonder if Target got hacked again...


I feel like some retailer had a problem with its registers last November or maybe the previous November where a similar thing happened... or maybe it was a website... I forget which retailer it was...

Recently, I went to Safeway and walked up to a self checkout that was vacant. It had no customer the entire 30-40 seconds that I was approaching it. On the screen it had the "please wait" and then showed a card transaction still processing. I scanned an item and the scanner beeped but nothing happened. The employee came up and said this self checkout is going really slow today and it is still finishing the last transaction.

I moved to another self checkout and about a minute later as I was finishing my transaction, noticed the receipt was printing over on the "slow" unit and it then reset itself back to the start screen.

[mjhale]

These systems are far from fool proof. It is interesting Target did not have a back up mode or offline mode that would allow transactions to keep processing. But that causes a lot of problems as inventory tracking, pricing, etc. would not be available. Also it is strange timing for this to happen, it does not seem like a particularly busy weekend like a holiday or something. I wonder if Target got hacked again...

I worked for Hechinger (DC area home improvement chain that got done in by Lowes and Home Depot) in the late 1980s and early 1990s. We had an IBM mainframe POS system with NCR registers that was notorious for going offline at random times and taking up to 60 minutes to come back online after a power failure. We didn't have a generator or battery backup of any kind for any of our computer systems.

What saved us back then when systems went offline was that we still manually priced almost every product in the store. The exceptions were building materials like lumber, bagged goods and stone and concrete products. Originally this was because Hechinger still used manually entered SKU numbers until 1990 or so when we finally got scanners. After that they tried removing the price tags but the customers complained loudly. Having the price tags on most products saved us when the systems when down because we could continue by manually entering the SKU or barcode once scanning started and the price. The only hold up was that credit card transactions over $100 (I believe) had to be called in.

Of course pricing every item in the store these days would be a near impossibility from a labor standpoint. I too have to wonder why Target did not have some sort of backup system or methodology to fall on to. For a large company like them not to have something - even a limited mode for common items - seems like bad IT planning. Target isn't a mom and pop store running on a shoestring budget. And lastly, I may be cynical, but when a company immediately makes a statement that there wasn't an attack a bit of my mind wonders. It may not have been a data theft attack but maybe Target was subject to a denial of service attack. We will never know for sure. But what I can say is that I happened to be out during the height of the problems. The shopping center near me has Target and Walmart. The Target (which isn't really that busy since the Walmart is a Supercenter) had a mostly empty parking lot but Walmart was packed to the gills like it usually is.

[BillyGr]

These systems are far from fool proof. It is interesting Target did not have a back up mode or offline mode that would allow transactions to keep processing. But that causes a lot of problems as inventory tracking, pricing, etc. would not be available. Also it is strange timing for this to happen, it does not seem like a particularly busy weekend like a holiday or something. I wonder if Target got hacked again...

It was noted in one of the stories that they had a similar issue EXACTLY 5 years ago (to the day), which was a few months after their major hacking issue.

Of course pricing every item in the store these days would be a near impossibility from a labor standpoint. I too have to wonder why Target did not have some sort of backup system or methodology to fall on to. For a large company like them not to have something - even a limited mode for common items - seems like bad IT planning.

Surprisingly, it seems many of their items come pre-marked with prices - store branded things and even seasonal items in name brands that must be packed specifically for them. Not everything, of course, but they do have some.

Of course, they could just provide each guest with paper and pencil and have them write prices of each item from the shelf, but maybe that would be too Mom & Pop (or at least small chain - a supermarket did that once when their power was out, using the limited backup power for cold stuff and a light or two in each aisle).

[rwsandiego]

...
Surprisingly, it seems many of their items come pre-marked with prices - store branded things and even seasonal items in name brands that must be packed specifically for them. Not everything, of course, but they do have some....

The vast majority of items other than clothing and some home decor are not pre-priced. Plus, pre-priced items that are on sale would not be pre-priced with the sale price, so it would be difficult to identify the correct price....[/quote]

...
Of course, they could just provide each guest with paper and pencil and have them write prices of each item from the shelf, but maybe that would be too Mom & Pop (or at least small chain - a supermarket did that once when their power was out, using the limited backup power for cold stuff and a light or two in each aisle).

In a cash-based retail environment this could work. However, when customers pay with gift cards, credit cards, and debit cards going completely manual simply does not work because authorization occurs via the POS system. Issuers and merchant banks are no longer set up to accommodate phoning in every transaction. Forgoing authorization can't occur, as they would not be able to submit the charges for settlement without the authorization code. There is a concept of offline mode, but not for the hundreds of thousands of transactions that would be submitted during an outage like this one.

In the end, chain-wide outages are extremely rare. Frankly, this is but a blip for Target. They will recover easily and quickly.

[Alpha8472]

My former boss used to work for a card processing company. McDonald's was a potential client, but his company decided against it. If the credit card systems at McDonald's restaurants failed for more than an hour, McDonald's said they would sue. McDonalds was too much trouble and not enough profit for that company.

However, there should be backup systems in place. A failure of this kind is probably due to hacking or sabotage from within. Either that or some very incompetent employees in the company or a subcontractor. Again, companies may never find out if it was hacking. They will describe it as an unexplained glitch or cover it up.

In 2017, Russian retailers suffered billions of losses in rubles as internet connected registers failed during the holiday season . The internet connected registers were required by law to cut down on tax avoidance.

[storewanderer]

My former boss used to work for a card processing company. McDonald's was a potential client, but his company decided against it. If the credit card systems at McDonald's restaurants failed for more than an hour, McDonald's said they would sue. McDonalds was too much trouble and not enough profit for that company.

However, there should be backup systems in place. A failure of this kind is probably due to hacking or sabotage from within. Either that or some very incompetent employees in the company or a subcontractor. Again, companies may never find out if it was hacking. They will describe it as an unexplained glitch or cover it up.

In 2017, Russian retailers suffered billions of losses in rubles as internet connected registers failed during the holiday season . The internet connected registers were required by law to cut down on tax avoidance.

I've been to more than a few McDonalds after midnight that claim their card systems are down and they are cash only. From here in Reno to even back in a busy DC suburb (Crystal City).

Most stores have back up card processing in place or if card processing systems go down (it does happen) they batch the transactions in queue then process when the system comes back up. This usually is only done for small value transactions as they aren't authorizing the transaction and some stores in high fraud areas will refuse to process any cards if they're offline. I witnessed Save Mart go offline multiple times as they were installing Chip and every time they would process as Credit only small value transactions (below $25 or below $50) then the charge would always show up the next day. You had to sign for the purchase and there was a notation on the receipt with a weird Auth number (AB123 or something) and that was how you knew it was offline. However Save Mart can actually process Chip cards when offline as well.

Also maybe a year ago came across a Shell Gas Station that was "offline." This was very weird as they accepted cards at the pump but not in the store... store was cash only. It was like this for a few days. The pumps would only allow you to pump up to $50, Rewards did not work, etc.

In the past stores could "dial out" if the online system was broken. I believe the chip cards probably make it so you can't process like that anymore (unless you want to take the fraud risk).

[BillyGr]

...
Of course, they could just provide each guest with paper and pencil and have them write prices of each item from the shelf, but maybe that would be too Mom & Pop (or at least small chain - a supermarket did that once when their power was out, using the limited backup power for cold stuff and a light or two in each aisle).

In a cash-based retail environment this could work. However, when customers pay with gift cards, credit cards, and debit cards going completely manual simply does not work because authorization occurs via the POS system. Issuers and merchant banks are no longer set up to accommodate phoning in every transaction. Forgoing authorization can't occur, as they would not be able to submit the charges for settlement without the authorization code. There is a concept of offline mode, but not for the hundreds of thousands of transactions that would be submitted during an outage like this one.

In the end, chain-wide outages are extremely rare. Frankly, this is but a blip for Target. They will recover easily and quickly.

Probably true, though one might suspect if they announced that their systems were down for an unknown amount of time and the options were to wait or be processed if you could pay cash and had the prices of your items available (which for many today would be even easier - just snap a photo of each item with it's tag on the shelf or sale sign that you could flip through to show the checker), many might choose to go for that option rather than waiting or just walking out not knowing when they could purchase items (and the store losing those sales completely).

[veteran+]

I was at the Mission Valley - San Diego store when this happened!

It was backed up a bit but management stepped in quickly and expedited things along pretty well.

Received FREE bags and $5.00 off purchase.

[wnetmacman]

Nobody's registers (or computers) are foolproof. This is a shining example, and it also shows which store managers were ready, and which weren't.

NCR is the register of choice for Target. Many of these systems run on Windows 7, surprisingly, but that wasn't the cause of both outages.

Specifically, Saturday's outage was from HQ directly, which was surprising in that they had nothing in place in the local store to back them up. Walmart's IS staff has each store scanning from their own database locally where only credit processing takes place outside the store. Target, OTOH, does it all in the cloud. The issue with this surfaced Saturday when stores couldn't scan items. Cashiers were scanning with cell phones in some stores to look up pricing online.

Sunday's outage was blamed on NCR; their payment processing systems went down. Target could still accept cash, but not electronic payments.

To me, Target spends more on IT than Walmart does, but Walmart seems to always be able to make gold from horse manure. You don't hear of this type of outage at Walmart, because they have multiple backups in place that cover this. Target either doesn't, or the safeguards didn't work Saturday and Sunday.

Full disclosure: IT guy who's done third party work in both company's stores.