Protx back up
Protx Apology Letter from Managing Director
The following letter was emailed to all Protex Merchants today, now that Protx are back up and have backed down on mandatory 3d secure on Maestro cards:
To all of our merchants I would like to sincerely apologise for the inconvenience that was caused to your business yesterday due to the extended system downtime on the Protx payment gateway.
I have always made it a principle that the Protx culture is one where we should be completely open and honest with our merchants over any system difficulties that we have experienced. I believe that this guiding principle is something that is greatly appreciated by our merchants even though any system downtime is always undesirable.
Many of our loyal merchants have remained with Protx for many years even when we have had times of great difficulty. I believe that our honesty in the level of service that we are able to provide has played a large part in this.
I would therefore like to take this opportunity to share with you the reasons that led to yesterday’s events so that you can understand the circumstances that led to our system outage and also the actions that we took to restore normal service to your account.
Since January 2007 Protx has been communicating with all of its merchants to advise them of the significant changes that were taking place in the payments marketplace and how this would affect their accounts.
The most significant change this year is the introduction of 3D Secure, which is similar to an online version of “Chip and PIN”. This is being introduced by the banks to help prevent/reduce fraud for e-commerce transactions. We had therefore been in discussions with the acquiring banks for many months prior to our systems upgrade yesterday to establish their position on implementing these changes, and other changes concerning the processing of Mail Order/ Telephone Order (MOTO) and pre-authorised transactions.
The system upgrade that was scheduled to take place between 6am-8am yesterday represented the largest system change that Protx have implemented since November 2005 and followed many months of detailed planning and testing. Our new payment gateway system has been designed to include all of the new bank rules for processing transactions.
However, it became apparent very soon after the live service was restored that the live service was not completing payment transactions in a short enough time period after we had received a response from the acquiring banks. This led to transactions queuing and eventually this overloaded our payment gateway system leading to the system becoming unavailable.
Protx had successfully and rigorously tested its new payment gateway software in a test environment and on a limited live setup. The issue that led to the extended downtime yesterday had not occurred at any stage during our testing.
After an exhaustive and detailed analysis of the messaging processes controlling communication with the banks my technical team were able to identify the root cause of the delay in processing responses to be at a very deep level within our database. It was eventually established that one table in the database had not been indexed properly and that this was the primary cause for transactions not completing.
Once we had identified and rectified the problem we were able to restore full service at around 2.30pm yesterday. It was my decision as CEO to resolve the system outage on the new platform and not to roll back to the old Protx payment platform which may have caused even more disruption to our merchants.
However, it then became apparent that many merchants had either not read or perhaps not properly understood the deadline date in the emails that we had sent to them over the last 6 months; regarding the changes that were being made to their accounts in respect of 3D Secure for domestic Maestro card processing and the need to have a separate merchant account for Mail Order/Telephone Order (MOTO) and e-commerce transactions.
Many merchants may have felt that the system outage lasted beyond 2:30pm because they had not applied the changes that we had informed them of over the last 6 months. As a result these merchants may have been unable to process domestic Maestro transactions and/or transactions through the Protx virtual terminal, which many of our merchants use to process MOTO transactions.
Earlier today my team and I were able to get definitive statements from the acquiring banks on how they would like to move forward in respect of 3D Secure registration and when merchants will be required to have separate merchant accounts for e-commerce and MOTO transactions.
In this respect a further e-mail has already been circulated to all of our merchants providing specific information regarding the acquiring bank that provides your merchant facility. I would urge all of our merchants to read this email.
I sincerely regret any inconvenience that any of our merchants have experienced through the delayed downtime whilst we were implementing our new payment processing software and the loss of any business that you may have incurred as a result of this downtime. I fully appreciate just how critical the payment process is for all of our merchants who operate online, but sometimes there are events that we simply cannot legislate for as there will always be differences between testing in a test environment and processing in a live environment.
I am pleased to report that the new Protx payment processing system is significantly faster and offers much greater resilience and robustness than its predecessor. Of course we all know the old adage about this being “early days” but all the indications from our monitoring are showing significant performance improvements. The systems upgrade we carried out yesterday was absolutely necessary for Protx to be able to continue to process an ever growing number of transactions.
I am embarrassed by the different and sometimes conflicting advice that many of you will have received from your acquiring bank and from Protx regarding 3D Secure registration and separate merchant accounts for MOTO and e-commerce transactions. I also realise that this must have been extremely frustrating and I would again apologise for any inconvenience that this has caused you.
My staff did their utmost to ensure that Protx met bank requirements but we found this to be a significant challenge in trying to legislate and manage the banks’ changing requirements where one rule would be applied for one merchant and another rule would be applied for a different merchant.
We have as a result learned some valuable lessons which we will use in the future to provide an even better service to you.
I also received criticism yesterday from many of our merchants in their belief that we had not communicated our system changes effectively and that we had not provided sufficient information about our system availability.
As an e-commerce business we communicate to all of our merchants primarily through the Protx website, where we post regular news items advising our merchants of important system changes.
We also send regular e-mail updates to all of our merchants to advise them what action they need to take. In addition there is also a live system monitor on our website which provides system status reports to alert merchants of any system difficulties.
Finally, since the implementation of our new telephone system last week, which we are still coming to grips with, we also provide system update messages via this medium and you should call 08701114455 and press option 5 to get a live systems update.
I would like to thank the vast majority of our merchants for their continued patience in waiting for a response from our sales and support teams. Our support and technical teams have received an unprecedented number of enquiries that has pushed our resources to the limit. Your patience is therefore greatly appreciated whilst we work through your e-mails and respond to you.