Northern Rock and online banking capacity

A

askew70

Registered User
Messages
152
BraveInca said:
- When the system is operating normally it asks for a different pair of characters each time. The fact that it is not doing this is a symptom of it being overloaded IMO.
Click to expand...

No it's not, it is a symptom of it being badly implemented. Every time you connect to the server, to login, it has to issue you with a prompt for your details - this obviously takes processing power so if you get a prompt then the server had the grunt to get this far in the process. It also has to select two characters to prompt you for, and it has to record which character positions those are to match against your full password later - this is the step at which it should nominate which two random characters which you are expected to supply, and the fact that it doesn't select random characters every times means that the selection process is not really random at all, which equates to weaker security because the system is badly designed and/or implemented and not because the server(s) doesn't have the resources to function.



Of course a bank can be expect to design systems that handle the numbers of customers it has. Anything less than that amounts to poor and shoddy service. NR have a system which is badly implemented either because the people they contracted to build it don't know what they are doing, or because NR are not willing to put up enough money to pay for a real and proper system (i.e. well designed and with adequate hardware and software to allow it to function well).

As an example of a system that works well, 24 hours a day, for many millions of people, look at Google. When was the last time that you got a timed out connection when you used Google's search facility. And that involves hefty database lookups which require significant horsepower (and/or an extremely well designed system). The Google system is so good, because if it weren't no-one would use it and Google would cease to exist. The NR system being bad, relatively speaking, reflects badly on NR.
 
Re: Northern Rock bailed out by Bank of England


I guess you don't have much experience of IT systems.

All IT systems are designed to handle expected volumes and peaks within a certain set of parameters. It would be ludicrous to pay for a system that could handle 75% of customers logging in at once when your normal volume is, say, 0.05 % of customers logging in at the same time.

Google is designed to handle Google's expected volumes, NR is designed to handle NRs expected volumes.

Ever heard of a Denial of Service attack? Where a hacker simulates many simultaneous http requests to bring down a website? This has happened in the past to Yahoo and Paddypower, and is the equivilant of what is happening to NR at the moment.

Lastly, even if it wasn't massively expensive to allow 100% of customers to withdraw at the same time, no sane bank would implement a system that could do that.
 
Re: Northern Rock bailed out by Bank of England

I'd like to know if the design of the system allows the same number of successful transactions to be completed when it's under high load. If it's poorly designed it could actually be processing fewer transactions than when it isn't under high load. This could account for the long periods during peak load when nobody reported getting logged in and completing a transaction. This explanation doesn't need a conspiracy theory to account for the reduced transaction processing rate.

As Askew70 said a well designed system like google's (where programmers are given the time and respect necessary to do the job right) reacts well under load by not locking out all users. Any well designed system also uses dimensioning rules for expected processing requirement rates so that costs may be balanced with benefits.
 
Re: Northern Rock bailed out by Bank of England

BraveInca said:
I guess you don't have much experience of IT systems.
Click to expand...

Oh, only about 15 years or so. Will that do, at all?


So, it is ludicrous to pay for a system that can accommodate even the majority (let alone all) of your customers at the same time? Someone better tell places like Tesco, for example, that they should reduce the number of cash registers in their stores, as Tesco seem to be under the impression that lack of resources in that area will result in their customers choosing to go elsewhere.

I wonder if you would be so forgiving of NR if you were a customer that dealt with them at a branch with one desk to service all customers, regardless of whether the number of customers was 1 or 1,000?

BraveInca said:
Google is designed to handle Google's expected volumes, NR is designed to handle NRs expected volumes.
Click to expand...

I take it then that if you are a customer of NR then you have always found their web service to be very speedy and responsive? Posts on this thread, and my own personal experience, suggest you would be in the minority if so.

BraveInca said:
Ever heard of a Denial of Service attack? Where a hacker simulates many simultaneous http requests to bring down a website? This has happened in the past to Yahoo and Paddypower, and is the equivilant of what is happening to NR at the moment.
Click to expand...

Ah yes, Denial of Service attacks. Yup, I know about those. Typically though, a provider of a service would design their system so that their own customers trying to avail of the service does not cause a Denial of Service. Usually the cause of a Denial of Service is expected to be a malicious or careless third party, not attempted valid use of the service as that would generally be considered shooting yourself in the foot.

Incidentally, defending against a Denial of Service attack is typically a lot more feasible than helping a poorly resourced/implemented system to limp along under load.

BraveInca said:
Lastly, even if it wasn't massively expensive to allow 100% of customers to withdraw at the same time, no sane bank would implement a system that could do that.
Click to expand...

To follow that line of thinking, if they had branches on the street, would NR be entitled to lock the doors and close up shop for the day if the queue of customers built up to more than, say, 50?
 
Re: Northern Rock bailed out by Bank of England

I can understand your anger askew70, but if you can find a bank with an online facility with the bandwidth that you require, I suggest that you bank with them, I am not so naive as to think that I am going to find one.

In answer to your last question, banks have been doing this for years.
I am not excusing it, just realistic. In the days when all retail banking had to be done in the branch, lots of branches would just close the doors early
because the queues inside were too long to be dealt with that day.

Different technology, same service.
 
Re: Northern Rock bailed out by Bank of England



Wow, you're really missing the point.

To follow your analogy, Tesco have, say, 1 million customers in Ireland. If there was news of impending food shortages and all one million customers decided that they needed to buy a month's food right now, would Tesco be able to cope?

Should they build capacity for 1 million customers in case this unlikely contingency should arise?

No, of course not. No business does. They build capacity for expected volumes plus reasonable contingency. This is what Google does, what Tesco does, what everyone does. If every user of the internet hit Google right now then the site would go down.

If your expected peak capacity is .05%, then you might build in contingency to be able to handle peaks of 100 times that (i.e. 5%).

Anyway, my reason for posting was simply to point out that there is a reasonable explanation why their systems would be unresponsive right now. Not trying to get under anyone's skin
 
Re: Northern Rock bailed out by Bank of England


I'm not angry, what would be the point of that. However, I am disappointed that a bank which sells a service based entirely on the premise that your account is available online at any time of the day, can't deliver on this.

The technology to provide a reliable and robust web-based service is not new, and Google is just one very well know example of that (and the load on the Google service is many many times greater than that on NR's web service). Unfortunately, as with many things, the true quality of the service only becomes apparent in times of quasi-crisis (i.e. in this case that manifests itself as heavy load on the server(s)), and the NR web service is proving itself to be extremely poor. They simply haven't invested enough in this core aspect of their service, be that an investment of time, money, and/or quality control.

Would the online service of other banks fare just as badly in the same kind of situation? Maybe. But the more important question is should customers of such a service be willing to accept poor quality of service and the answer to that must be no. A poor quality service is a poor quality service regardless of whether it is face to face in a branch, or on-line.

Unfortunately, the issues with NR's service seems to be tainting peoples' view of online banking services generally, if some of the post in this thread are anything to go by. That is a shame because NR's service, to me, falls short of even attaining the middle ground of the quality range and is therefore not a good yardstick by which to measure similar services by other parties.
 
Re: Northern Rock bailed out by Bank of England


If you build a system capable of handling only 5% of your customers, yet continue to advertise to all existing and potential customers that this service will be available 24 hours a day, 7 days a week, then you should be penalised for false advertising.

And yes, there is a reasonable explanation for why their systems would be unresponsive right now - their system was simply not designed to handle the load. This happens with a lot of websites, but you don't expect it to happen with a website whose service is so valuable and important to so many people.

To take another example, as you don't seem to like my Tesco analogy: If a plumber installed a toilet in your house that simply failed to operate after it reached a limit of 5 flushes in any one day, then I suspect you'd be a wee bit upset. You can choose to be selective about which services you expect quality from, but I prefer to be more consistent.
 
Re: Northern Rock bailed out by Bank of England


I can have a crack at that one too

If everyone in your house had a bad case of the runs and needed to use the toilet at exactly the same time, but they couldn't, would you blame the plumber? Would you pay for and maintain individual toilets for each person in case this contingency should arise? Most people don't because they are happy to have enough toilet capacity for normal demand.

Okay, enough! I agree that its frustrating, but so far they haven't failed my service expectations. However, the situation continues for much longer then my opinion will certainly change. I'm willing to give them the benefit of the doubt for a couple of days in the current circumstances.
 
Re: Northern Rock bailed out by Bank of England


Service expectations should be based on what the service claims to offer. NR offer 24x7 access, and use this as a strong (and perhaps even the only) selling point for their service, but when tested they can't provide this. I am surprised that anyone considers this reasonable, at any time. It is not beyond the reach of a company with sizeable funds available to build a robust system to cater for hundreds of thousands, and perhaps millions, of customers, but it requires the commitment of the company offering that service, and NR's service has demonstrated their lack of commitment in that regards.

I managed to finally login a few minutes ago, for the first time in the last 42 hours. That, to me, falls very far short of a decent service.
 
Re: Northern Rock bailed out by Bank of England


Building the kind of capacity to cope with an extremely rare event such as this would be a ludicrous waste of money. It's unfair to expect Northern Rock to have such capacity available, and it isn't the least bit surprising that the system is next to impossible to access in the midst of this present panic. I have always felt though that the NR online system was rather sluggish under normal conditions and certainly this is something that could have been improved. But I regard that as a separate issue from what's going on right now. Even if normal service was more responsive, it's hard to see how it would make much difference under the kind of demand it's facing at present.

This is getting annoying now. I intended to withdraw some money for the week after next and it looks like I'm gonna have to waste time and endure frustration battling with the online system sooner rather than later if I'm gonna be sure the money's out when I want it. It really is extraordinary the level of panic this has produced. I can understand if all someone's life savings is in NR, or they're above the protection limits and want to move some of the money for peace of mind, but there are definitely people overreacting to all of this and making matters worse for everyone else.
 
Re: Northern Rock bailed out by Bank of England

On-Line Transaction Processing (OLTP) is a complex business. As the good general points out, many systems operate poorly when stressed. As an example of this with NR, I found last night that I was getting timeouts (the service unavailable message) after I had signed into my account. I found that (using firefox), if I reloaded the frame, I could get back to where I was (i.e. still signed in) as I posted earlier.

I would agree that, in particular, their security servers are undersized and look like serious bottlenecks at high volumes. It also appears that they are using multiple security servers that are not correctly scaled in size (if you get past the first one, you should be able to get past the second one easily and you should then be able to conduct transactions on your account without further timeouts). In other words, it looks like there is no capacity lock on the system - it doesn't shutdown access to new traffic when volumes are high, it just craps on everyone (whether trying to enter the system or already on it).

I don't believe that this is either intentional in design or a response to the current situation, I think the system is just badly designed from the outset.

Google is not a comparable operation, IMO, as it does not require security authentication. A better example would be if amazon had a massive sale on. I doubt if it would cope any better. As other posters have pointed out, ticketmaster have struggled even when they know there is exceptional demand coming for a particular event. Contrast this with the airline industry who have been running OLTP systems for the last 40 odd years and see that difference when Ryanair have a sale. Experience counts and the banks are novices at this.
 
Re: Northern Rock bailed out by Bank of England


You are making the same large assumptions that several others seem to be making.

For one thing you are assuming that building a system to cater for heavy load is extremely expensive. It can be, but you really only start to hit expensive solutions once you have exhausted the potential of what you already have. From what you say, you seem to be assuming that NR's system is bursting at the seams because the existing hardware and software is incapable of coping, and therefore more investment is required. Another possibility however is that the hardware and software they have is not the issue (yet) because the basic architecture of the solution is at fault. I have seen at least one appallingly designed web-based solution implemented by a financial institution where they threw money at the hardware and software, but the basic design was always going to result in failure of the service before any single machine even broke a sweat. In that case they didn't need more money to make a better solution, just more thought.

For another thing, you are assuming that massive numbers of NR customers are hitting the web service at the same time. That may be the case, but none of us know that for sure. It could well be that the system is so bad that it flounders even if 20 people access it simultaneously. In any case, any system which is well designed should "fail" gracefully, rather than providing the illusion of working until your connection eventually times out after several minutes.

Ultimately, as with most things, customers will end up with the service they deserve. If people are willing to accept an online service of the standard of NR's, without complaint, then there is no reason for any NR competitors to go to the trouble of building a better system. If enough people deem the recent performance of the NR online service as perfectly reasonable, then we are all doomed to crap online banking for some time to come.
 
Re: Northern Rock bailed out by Bank of England


The process of accessing Google is certainly different in that it doesn't involve authentication (although Google Mail is a heavily used service that I have known to be problematic on only a handful of occasions and even then only for a minute or two at most). Having said that, the Google service does tailor itself based on the results of some checks against your IP address, for example, so it does carry out some processing before it presents you with the search page. Also, the Google service does involve running searches on massive databases, and authentication can be seen as essentially just a database search for a user's details (followed by a comparison against what the user supplied).

However, implementation details aside, the main similarity between Google and NR's online service is that the success/survival of one arm of each company depends entirely on providing an online service that will entice customers in rather than scare them away. Google practically built their service from scratch to produce a very impressive solution, whereas NR seem to have thrown a few things together without much consideration for how it would fare under pressure (which many companies are equally guilty of, of course). Hopefully NR, and other banks, will learn from this recent clear demonstration of this mistaken approach. By comparison, Google have always known that their service being unavailable to a lot of people for several days in a row will lose them customers permanently, so they specifically designed their system to minimise this possibility, and very effectively too.
 
Re: Northern Rock bailed out by Bank of England


I made a withdrawal on Saturday, which today shows as having been processed. Mind you, the withdrawal still appears in my list of pending transactions, but as I have no faith left in their online service (as distinct from no faith left in the bank itself), this doesn't unduly worry me.

In fact, due to the persistent difficulties that I have had in accessing my online account since Thursday night last (and some people report problems back as far as the previous weekend), I decided to withdraw the rest of my money today and close my account. That didn't go smoothly either (having completed the withdrawal process I was thrown right back to step 1 again and had to repeat the withdrawal), but it now shows as a pending transaction.

Despite the views of some people here that this level of online service is only to be expected, I completely disagree. This situation would obviously push anyone's online service pretty hard, but NR's service has spent the last few days compeletely unusable, for the most part. I am repeating myself here, but this technology is not rocket science, it is well understood and capable of being used to provide a very good service for those companies for whom quality of service is important. Based on the shoddy behaviour of NR's authentication process, and the fact that the online service itself is simply unreachable most of the time for several days now, this particular service has shown itself to be sub-standard.

I have drawn the comparison to Google's search service before and I will repeat that again too - if the Google service was unusable for an hour or two (let alone several days), people would be annoyed and would flock to an alternative searcing service, and would possibly complain to others later that Google's quality had gone downhill. And that is a free service to most people who use it. Why should we be so forgiving of the online service of a bank which we fund with our own cash and which has potentially serious implications for us when it is unavailable?

Just to be clear, I view the bank service and its online service as separate entities. My decision to close my account today is motivated more by my dissatisfaction with the online service than anything else, as the new assurances from the BoE seem to me to be as good as anyone could have expected. I just don't wish to continue to fund, or be a frustrated user of, an unsatisfactory online service when I know it can be done much better.
 
Re: Northern Rock bailed out by Bank of England

Askew70 - having disagreed with you initially about the comparison with google/the difficulties of OLTP, I now find myself agreeing with you. This is day 5 and the service has not significantly improved. I would expect any reasonably IT department to have made improvements by now. As you say, the lack of responsiveness to the problems (or quite probably the badly constructed system that makes it difficult for them to respond) diminishes confidence in their ability to conduct day-to-day internet operations.

No doubt businesses all over the world are reviewing their models and peak capacity expectations, so maybe some good will come of it?
 
Re: Northern Rock bailed out by Bank of England

I don't think this is being fair on NR. No secure webservice could maintain its QoS given the flash crowd scenario that NR has experienced these last days. The compute time required to negotiate SSL handshakes for all of NR's customers runs into the order of 72 hours alone. Not to mention that the computation time needed for the DB back end would be several times that. The cost of over provisioning for that level of a spike makes no economic sense.
 
Re: Northern Rock bailed out by Bank of England


The expense in establishing an SSL connection is on the initial connection (verification of security certificate, negotiation of encryption keys, etc.). Once you have an SSL connection in place, which is before you are prompted for your account/login ID, then subsequent connections are pretty "cheap" in terms of server resources. At that stage, database access becomes the next expensive operation, and given that database technology has existed for a long time that is capable of handling load very well, there is no reason for this to be an insurmountable hurdle by any means either.

The nature of the failure of the NR web service right now is probably compounding their problems. Once you get an SSL connection established, to proceed with the authentication stage, more often than not authentication fails. If you subsequently quit that window, you lose your SSL connection and have to establish a new one from scratch, leading to greater load on the server. The success of people logging in through refreshing the existing window, with SSL connection already in place, might be down to the fact that these re-fresh connections don't add to the "SSL load" on the server.

People have referred here to the likes of Amazon and Ticketmaster as examples of services that might well fare badly in this situation too. I have had poor experiences with the Ticketmaster site, and am no fan of it, but I haven't experienced it being unusable for days at a time. I have never experienced significant problems with any of the other heavily used online services that I make use us (Google, Yahoo, Amazon, eBay, etc.). The difference is, I believe, that those companies give adequate importance to the quality and reliability of their online services, whereas NR clearly has not done so.

I can only hope that this issue will make other companies sit up and take notice that their online service is important. There are many, many, companies that have an online presence which they seem to take no interest in - symptoms of this range from lack of SSL connections, to sensitive information (such as credit card info) being transferred in the background via unprotected e-mail. In the worst cases, some of those services seem to me to verge on the criminally negligent, and will likely remain haphazardly implemented until there is pressure to improve. While I have concerns about the NR authentication process, it is the poor (lack of) availability of the service that is currently the biggest issue, and there too companies will feel no pressure to improve that aspect of their service if people deem current performance acceptable.
 
Re: Northern Rock bailed out by Bank of England

Nothing is insurmountable but the cost of provisioning for a system that could handle so many transactions at the same time just does not make sense economically. I'd rather see my bank put its funds elsewhere than have it wasted on resources for crazy "what-if" scenarios.

Of course there is room for improvement on QoS and reusing SSL connections is a technique that can speed things up. This could compromise security somewhat which means it's use in banking web services is limited. Higher security obviously has to take preference over higher quality of service in the banking world.

I don't think that the likes of ticketmaster or amazon have yet encountered a situation where up to 1.5 million customers were all trying to log in and perform transactional operations at the exact same time, and over such an extended period. I would imagine they would experience a lower QoS too given the circumstances.
 
The Google comparison is invalid. Google's income scales with the number of simultaneous users they can serve, so they have a financial incentive to ensure they can serve as many simultaneous users as possible. Banks do not have the same financial incentive, which makes it uneconomical to overprovision for the worst case scenario. Of course NR could provide an online service capable of handling this load, but very few customers would accept the costs of such a "gold-plated" service. If anything (as pointed out earlier in the main thread) there is a financial disincentive to build systems that facilitate a run on a bank.

Additionally, banking is transactional, and search is not. Google can copy their read-only search database to scale indefinitely (often with inconsistent distributed copies of the database). Banks need to store a consistent record of transactions on an account. Google's approach to scaling more transactional services (such as Google Mail) has been far more cautious, and less reliable.

I've never been particularly impressed by NR's online banking, but it does the job. The main fault with NR is that the service didn't degrade gracefully, though they seem to be taking steps to address that. I think it comes back to cost, which is reflected in interest rates. Rabo's online service has far more bells and whistles, but a lower interest rate overall. Whereas First Active's new online savings account has a higher rate, but by all accounts worse service and minimal online functionality.
 
You must log in or register to reply here.
Menu
Log in
Register