Understanding what went wrong during the BoI online/ATM glitch

Brendan Burgess

Brendan Burgess

Founder
Messages
52,184
Moderator's note: This thread is for trying to understand exactly what went wrong. Keep the more discursive issues in the other thread.


Just checked with Bank of Ireland.
BoI customers who also have Revolut cards could have stolen up to €1,800

1) With their Bank of Ireland card, they withdrew €500 from the ATM.
2) Using Revolut, they sucked €1,300 from their BoI account via their BoI debit card
3) Yesterday, they withdrew the maximum €1,000 from Revolut
4) Today, they withdraw the €300 balance.

However, there was no hurry in using the Revolut card yesterday. Once the money is in their Revolut account, BoI can't touch it.

Brendan
 
Last edited:
Brendan Burgess said:
Just checked with Bank of Ireland.
BoI customers who also have Revolut cards could have robbed up to €1,800

1) With their Bank of Ireland card, they withdrew €500 from the ATM.
2) Using Revolut, they sucked €1,300 from their BoI account.
3) Yesterday, they withdrew the maximum €1,000 from Revolut
4) Today, they withdraw the €300 balance.

However, there was no hurry in using the Revolut card yesterday. Once the money is in their Revolut account, BoI can't touch it.

Brendan
Click to expand...

What does 'Using Revolut, they sucked €1,300 from their BoI account.' actually mean? If this is a standard SEPA Direct Debit from their BOI account, It will take a couple of days to process. Revolut may well credit their account instantly, but when the debit hits BOI, it will will bounce if there are insufficient funds. When Revolut get notification of 'insufficient fund' they will reduce their balance accordingly. With SDD it will all take several days to fully process. The net result is balances will auto correct. However, if a customer pushes the 1,300 back to BOI (before hand) they may end up +1,300 in BOI and -1,300 in Revolut.
 
Towger said:
What does 'Using Revolut, they sucked €1,300 from their BoI account.' actually mean?
Click to expand...

I use Revolut with AIB and it comes out of AIB into my Revolut account simultaneously and I can withdraw it.

I assume it's the same with BoI?

I very much doubt that Revolut allows people to take out money which is not there and then spend it via Revolut. Only for Revolut to find a few days later that there is no money.

Brendan
 
Towger said:
Irish banks don't support SEPA Instant Payments. If the AIB account is being debited instantly, I assume it is a Debit Card payment rather then a SDD.
Click to expand...
This.

What likely happened is that BOI had a systems failure causing their payment processor (Total Systems?) to activate STIP - STand In Processing.
While in STIP, the payment processor might authorise transactions up to a certain limit agreed between BOI and the processor, in this case, likely €1,000.

Customers might have been able to make purchases beyond their balance, up to €1,000, but for some, using their BOI Debit Card to pull €1,000 into their Revolut, and then withdrawing that from an ATM, allowed them instant access to hard cash instead.

STIP is a useful function for a bank that is prone to outages, planned or otherwise, allowing brief systems failures or maintenance to fly under the radar for a lot of their customer's card transactions.

Reconciling a long STIP period is likely a painful piece of work, I would not have faith that BOI have automated this, many BOI employees had a bad day today.

For those that did withdraw €1,000 via their Revolut card, would they have hit Revolut ATM charges 2% for anything above €200, so €16 in this case. Some hand money for Revolut.
 
@oketc

That is a great explanation - thanks.

Can I just clarify. Is the following correct?

Because of the online problem, people were unable to access cash at ATMs.
But BoI overrode this to facilitate customers.

Brendan
 
Brendan Burgess said:
I use Revolut with AIB and it comes out of AIB into my Revolut account simultaneously and I can withdraw it.
Click to expand...

OK, I just tested it there and this is not correct.

I went onto Revolut and added €10 using what appears as "Easy Bank Transfer" on Revolut.
It brought me to my AIB account and I had to log in and it was a prefilled normal Sepa payment
It said"If done before 2 pm, it will be in your bank account today. If done after 2 pm it will be in your bank account tomorrow".

It has not appeared instantly in my AIB account as I thought it did.

So what did the BoI & Revolut customers do?

Did they go to their Bank of Ireland app and push the money into their Revolut account?
How did this happen instantly?

Brendan
 
Brendan Burgess said:
OK, I just tested it there and this is not correct.

I went onto Revolut and added €10 using what appears as "Easy Bank Transfer" on Revolut.
It brought me to my AIB account and I had to log in and it was a prefilled normal Sepa payment
It said"If done before 2 pm, it will be in your bank account today. If done after 2 pm it will be in your bank account tomorrow".

It has not appeared instantly in my AIB account as I thought it did.

So what did the BoI & Revolut customers do?

Did they go to their Bank of Ireland app and push the money into their Revolut account?
How did this happen instantly?

Brendan
Click to expand...
I'm pretty sure it depends on how you have set up your Revolut for topping up. You can either do it via a SEPA payment or via a POS transaction using your debit card. What must have happened is that the BOI accounts appeared to have an available balance that would allow for cash withdrawals and card payments over what was available etc. so anyone funding their Revolut account via a BOI debit card could top up instantly using the magic money.
 
Brendan Burgess said:
@24601

Excellent.

I have just sucked €10 out of my AIB debit card and it appeared instantly in Revolut.

Brendan
Click to expand...
That STIP theory from @Okokokoknic seems the most plausible so. People basically had a temporary overdraft of something like €1,000 for a period, allowing any POS transactions and/or cash withdrawals up to that amount. Revolut were the primary platform facilitating the fraud/theft (other than ATMs) as they appear to be the only bank that allows POS payments into a current account rather than SEPA transfers. Of course, SEPA instant could have been exploited in the same way, but I don't think that's widely available here.
 
Stand in processing might well explain why some transactions were allowed but that just shows that BOIs communications that day about technical issues with their online banking was a complete fabrication.

Online banking going down does not impact ATMs or POS Transactions. It was obviously a much wider and complete systems issue.


Mad thing is that BOI are very unlikely to ever publicly reveal what happened.
 
Last edited:
Sunny said:
Stand in processing might well explain why some transactions were allowed but that just shows that BOIs communications that day about technical issues with their online banking was a complete fabrication.

Online banking going down does not impact ATMs or POS Transactions. It was obviously a much wider and complete systems issue.


Mad thing is that BOI are very unlikely to ever publicly reveal what happened.
Click to expand...
Yes, this makes sense. I routinely top up my Wise USD balance from my US-based SunTrust account. I can transfer up to 15k at a time which instantly lands as cleared funds in Wise (ie transferable elsewhere in any currency or even as cash via Wise card). But notice that the debit takes days to appear/update in the originating SunTrust account. This time lag technically allows me access to the same money in two accounts at once. Easily exploitable if I were so inclined.
 
Last edited:
Okokokoknic said:
Customers might have been able to make purchases beyond their balance, up to €1,000, but for some, using their BOI Debit Card to pull €1,000 into their Revolut, and then withdrawing that from an ATM, allowed them instant access to hard cash instead.
Click to expand...
Thanks for sharing your hypothesis, which sounds credible.

One question that occurs to me, in this scenario why would the focus have been on BOI ATMs ? Once the money had been transferred to a Revolut account/card, surely it would have been accessible from any ATM ?
 
24601 said:
That STIP theory from @Okokokoknic seems the most plausible so. People basically had a temporary overdraft of something like €1,000 for a period, allowing any POS transactions and/or cash withdrawals up to that amount. Revolut were the primary platform facilitating the fraud/theft (other than ATMs) as they appear to be the only bank that allows POS payments into a current account rather than SEPA transfers. Of course, SEPA instant could have been exploited in the same way, but I don't think that's widely available here.
Click to expand...
People could have also paid off other Credit Cards e.g. AvantCard using the same process. Anything that used a POS payment I guess.
The money never actually needed to be taken out of an ATM, people just did so, I would imagine, because they feel in control of it.
 
Freelance said:
Thanks for sharing your hypothesis, which sounds credible.

One question that occurs to me, in this scenario why would the focus have been on BOI ATMs ? Once the money had been transferred to a Revolut account/card, surely it would have been accessible from any ATM ?
Click to expand...
People wouldn't need to use BOI ATMs in this scenario, I'm fairly sure I saw a clip doing the round of Gardai standing in front of ATMs from a number of banks.

The focus would have been on BOI ATMs, as people saw the underlying issue was a BOI one, but ultimately, the 'free money' could be accessed via your card, wherever you chose to do so.
 
Brendan Burgess said:
@oketc

That is a great explanation - thanks.

Can I just clarify. Is the following correct?

Because of the online problem, people were unable to access cash at ATMs.
But BoI overrode this to facilitate customers.

Brendan
Click to expand...
The online/mobile user interface is merely a front door that customers access. The problems would have been in the back end systems, which is where the payment authorisations relating to card transactions come into it.

The front end systems likely would have slowed at first, struggling to get a response from the back end.
This would cause a backup of customers to keep trying to access the front end systems.
People keep retrying, putting the front end under strain.
Someone then complains on social media that the front end is not working, anyone that sees it immediately pulls out their phone out of curiosity to see "is it broken for me too", this can have the effect of knocking over an already stumbling front end, so it doesn't even present a login page etc.
 
On 30 November 2021, the Central Bank of Ireland (the Central Bank) reprimanded and fined Bank of Ireland €24,500,000 for failures to have a robust framework in place to ensure continuity of service for its customers in the event of a significant IT disruption. These IT service continuity deficiencies were repeatedly identified from 2008 onwards but due to internal control failings only started to be appropriately recognised and addressed in 2015. The steps taken to address the deficiencies were completed by 2019.

It would be very interesting to see a full report on what happened. From the scant evidence available, it appears that there was more then one issue (1) initial failure somewhere in the BOI IT environment (2) a failure in the online systems (3) a failure in the ATM environment. It may well be some of the issues were consequences of the responses taken to the initial failure, as is often the case. There has also been talk that the bank overrode part of the automated shutdown in order to allow ATM transactions to continue, possibly without understanding the consequences. The CBI has published two relevant Guidance Documents, the 2016 Cross Industry Guidance in respect of Information Technology and Cybersecurity Risks and the 2021 Cross Industry Guidance on Operational Resilience. In light of these documents, and in the context of the 2021 fine, I would love to see the CBI insist on a full root cause review, and also a "lessons learned exercise" as called for in the Operational Resilience document. And also to understand whether or not the alleged remediation of the issues in 2019 was sufficient.

Of course the next obvious questions is whether to not BOI was just unlucky to be caught out, and whether or not the other players such as AIB, PTSB, CU in particular are any better positioned. I'm sure the CBI are crossing their fingers and saying their prayers and hoping against hope that there won't be another failure like this anytime soon. Or maybe they are actually doing something proactive about it, who knows ! After all, it's only money.
 
BOI outsourced most of their IT in 2000 long before it was the norm. They started out with Perot Systems, who I think were taken over by HP, then they went with IBM, last I heard it was a company called Kyndryl. Apparently the latter laid off a load of technical staff a few months ago, and this included a lot of the staff working on operational matters.
Without knowing the detail of the problem I cannot speculate, but am told that the lay off of the Kyndryl contractors would have been a major factor in recovery time being very slow.
 
@Iff12 You missed a few of the twists, turns and dead ends on the tortuous trail that is Boi’s never ending descent into IT hell !

The Perot stage was actually a JV between Perot and BOI. They had notions of reselling whatever they did for BOI across Europe. That had a messy end in 2002 when AIB and BOI announced that they were to merge their IT departments and spin the JV off as a standalone operation with I think 700 employees. That fell apart during the negotiating stage, however unfortunately BOI had given notice to Perot by then. So the next step was to jump to HP in 2003 on a seven year contract. That all happened on CIO Cyril Dunne and CEO Mike Soden’s watch, the latter having his own issues with BOI’s computer systems. BOI subsequently switched to IBM in 2010. Accenture have also had a long-standing outsourcing involvement with BOI, for purchasing and training from way back and this was extended considerably around 2014 to cover IT and “change management”. There is plenty more in terms of BOI’s IT governance shortcomings, both before 2000 going back to BOI’s ICL days, and in more recent years.

This and plenty of other twists and turns have brought BOI and its IT operations to where they are today. The €24,500,000 which the CBI fined BOI for IT Service Breaches in 2021 is indicative of just how bad things were/are.
 
This was more the STIP kicking in as a previous poster mentioned, although I do believe it probably did kick in. If it did, then potentially the banks losses will be even greater since card payments were working fine so card transactions in POS or online that should have been rejected were quite possibly accepted. Open door for fraudsters who if they were aware of what was happening, could have been hammering online retailers with every BOI card number they could find

The fact that online banking and the app were down suggests more to this, unless the bank did it deliberately to minimise the damage
 
thedaddyman said:
This was more the STIP kicking in as a previous poster mentioned, although I do believe it probably did kick in. If it did, then potentially the banks losses will be even greater since card payments were working fine so card transactions in POS or online that should have been rejected were quite possibly accepted. Open door for fraudsters who if they were aware of what was happening, could have been hammering online retailers with every BOI card number they could find

The fact that online banking and the app were down suggests more to this, unless the bank did it deliberately to minimise the damage
Click to expand...
Agreed there is more to it with front ends going down as well.
The front ends could have been brought down by the volume of customers checking them, BOI appear to have a DDOS service in front if them which could have started dropping the traffic.

Alternatively it could have been a wider issue affecting wherever BOI host their front & back ends, assuming all were hosted at the same site, or multiple sites affected by one common issue.
 
You must log in or register to reply here.
Back
Top