Bradford POP - fibre circuit failure

Fri Dec 23 13:52:55 2011

We are currently experiencing a loss of service to our POP in Bradford.

Fault has been raised and we will post more updates as they come in from the telco.

Fri Dec 23 23:25:44 2011

Major service outage in Bradford area caused by vermin eating through a major BT trunk fibre cable. Cabling team on-site pulling new cable length. Splicing team will be working through the night to restore service.

Sat Dec 24 14:10:29 2011

The service was restored at 13:36 by BT. Service is still at risk while BT is carrying out work.

Shared Unix Hosting - DDoS

Tue Dec 13 15:22:07 2011

One of our unix shared web servers is currently experience abnormally high load due to a denial of service attack targeted at a customer site.

Our engineers are currently working to resolve the issue but customers on the affected machine may notice degraded performance until the work is complete

Wed Dec 14 11:35:20 2011

This problem has now been resolved

SurfProtect/Web access

Fri Nov 25 11:00:26 2011

Due to an extraordinary load on our SurfProtect system, our engineers have brought forward the planned comissioning of additional servers into the system.

Unfortunately this has caused instability in the SurfProtect service which has affected all of our customers using SurfProtect. Our engineers have been working to correct the problem this morning, and as of 11am we believe this issue to have been resolved.

During the work, one of the servers had to be rebooted as it became unresponsive, unfortunately it did not come back. We have dispatched an engineer onsite to investigate the issue.

This failure is affecting our customers on BT DSL lines, other customers should find that services are working as expected.

We will continue to pro-actively monitor the service throughout the course of today, and apologise to all customers affected for the disruption this has caused.

Fri Nov 25 16:11:29 2011

Normal service has now been restored. Anybody experiencing further issues should contact support on 0845 145 1234

IMAP

Tue Nov 15 10:05:18 2011

We currently have an intermittent problem affecting IMAP. Some customers may be temporarily unable to read mail for the moment.

Our engineers are aware of the issue and are currently working to fix it. Further updates will be posted as they become available

Tue Nov 15 10:14:13 2011

This issue has now been resolved. The IMAP service had started to reject new connections and a reboot of the software has restored normal operation.

Mail

Wed Oct 26 15:10:08 2011

We are currently experiencing a hardware issue with one of our mail filestores.

Recent firmware updates have not fixed the issue so we will be migrating affected customer domains onto a new server.

Core switch in Manchester

Mon Oct 24 13:44:53 2011

We are seeing problems with a core switch in Manchester, which is sporadically rebooting. Our engineers are currently en route to replace the faulty hardware and we will post further updates as they become available.

Affected services include customer leased lines running through Manchester, DSL and mail services.

Mon Oct 24 16:51:22 2011

Faulty switch will be replaced between 5:30pm and 6:00pm.

Leeds - multiple (Openreach) fibre circuit failures

Wed Sep 28 15:15:23 2011

We are currently experiencing a loss of service on a number of customer fibre circuits (Openreach EAD/WES) that terminate in our Leeds Central POP. We suspect that a major trunk cable has been accidentally cut but Openreach cannot yet confirm this. Faults have been raised on all affected circuits and we will post more updates as they come in from the telco.

Wed Sep 28 21:31:53 2011

Openreach engineers have been performing OTDR (optical time domain reflectometer) tests from this evening and have identified a break 109m from the datacentre in a blown fibre bundle. A specialist team has been dispatched from York to blow a new bundle and re-splice affected circuits. Current estimated resolution time: midnight (00:00)

Thu Sep 29 00:13:08 2011

We are glad to report that the Openreach engineers have hit their target fix time of midnight. A new fibre bundle has been blown in and affected circuits re-spliced. We have confirmed that all circuits are now operational and will be monitoring closely for the rest of the night.

Leased lines

Tue Aug 2 17:30:27 2011

A fault on one of our core switches caused loss of service for some leased line customers at around17:20 today.

Tue Aug 2 17:31:15 2011

Our engineers restored service around 5 minutes after the problem occured

SQL server issue

Fri Jul 8 15:21:51 2011

Following an issue with MySQL on mysql2.exa-networks.co.uk, we performed emergency maintenance and gracefully rebooted the server.

The server did not come back and our engineers are now investigating the issue.

The fault will be affecting our webmail users as well as the websites using this database.

We expect to post an update to this problem at 15:40

Fri Jul 8 16:45:28 2011

Our engineers have restored service. Maintenance will need to be performed next week as part of an update to the service

Unixweb 1

Wed Apr 13 14:24:23 2011

Our shared web hosting server Unixweb 1 has stopped responding to requests. This appears to be a software malfunction, and we are currently rebooting the server to restore functionality.

Update: The reboot was successful, and Unixweb 1 is now accessible as normal. We are currently investigating the cause of the outage, which we still believe to be a software issue.

SurfProtect

Wed Mar 30 15:47:19 2011

The SurfProtect availability problems have now been solved.

We have identified an issue with the squid proxy software and written a fix that has now been applied to abate the problem.

While investigating, we also identified bottlenecks in the SurfProtect classifier code and improved the caching of requests.

We have also added additional monitoring of processes on the SurfProtect cluster to detect early signs of any future issue.

SurfProtect

Wed Mar 30 12:30:41 2011

Update 3: We have corrected one problem with the underlying proxy software which was behaving erratically under high loads. However, as we are still experiencing high usage on SurfProtect we are also working towards performance improvements on the SurfProtect code.

Service should be restored, but due to the nature of the load and the current ongoing investigations the SurfProtect service may be intermittent throughout the afternoon.

A further update will be provided later this afternoon, at which point we hope to have implemented a fully working solution to the problem.

SurfProtect

Wed Mar 30 11:32:20 2011

An update will be provided in 30 minutes.

Update 1: Our engineers are still working on the problem, and are attempting to alleviate the current instability using multiple methods.

Update 2: An initial patch to the underlying proxy software was unsuccessful. Further modifications to the SurfProtect code are in process.

A further update will be provided in 30 minutes.

SurfProtect

Wed Mar 30 10:50:16 2011

We are currently experiencing a problem with our SurfProtect cluster, as unusual load patterns appear to be causing instability.

Users of SurfProtect will be experiencing intermittent web access until we can rectify the situation.

An update will be provided in 30 minutes.

Update 1: Our engineers are still working on the problem, and are attempting to alleviate the current instability using multiple methods.

Update 2: An initial patch to the underlying proxy software was unsuccessful. Further modifications to the SurfProtect code are in process.

A further update will be provided in 30 minutes.

SurfProtect

Wed Mar 30 10:15:08 2011

We are currently experiencing a problem with our SurfProtect cluster, as unusual load patterns appear to be causing instability.

Users of SurfProtect will be experiencing intermittent web access until we can rectify the situation.

An update will be provided in 30 minutes.

Wed Mar 30 10:47:47 2011

Update 1: Our engineers are still working on the problem, and are attempting to alleviate the current instability using multiple methods. A further update will be provided in 30 minutes.

Office telephones down (0845 145 1234)

Wed Mar 9 14:33:15 2011

We are currently experiencing an ISDN fault which has brought down our office telephone lines.

Customers attempting to contact us on 0845 145 1234 will be unable to do so until the fault is rectified. Please call our alternative number 0845 056 9702, which will be relayed to us via mobile.

We can still be contacted as usual via email.

Wed Mar 9 14:53:32 2011

The ISDN fault which brought down our office telephone lines has been resolved. Customers attempting to contact us on 0845 145 1234 will be able to do so as normal.

Network monitoring failure

Tue Mar 8 17:23:31 2011

Due to a disk failure in the Cacti NMS machine, historic data from this network monitoring service has been lost.

This will not affect customers who have access to our JFFNMS system, or customers who do not access our NMS systems.

We are currently working on restoring the operation of the Cacti system; data will be then be available starting from the time at which this is achieved.

ADSL Connectivity

Wed Nov 17 11:18:47 2010

We are currently experiencing an problem with connectivity to BT that is causing a loss of service for all customers of our adsl business and sdsl business products.

Our engineers are working to correct the issue and we will post further updates as soon as they become available.

3G Access

Mon Nov 8 10:34:03 2010

A problem within 3's network is currently preventing 3g sim cards from connecting to us. Our engineers are speaking with 3 to and we will post an ETA for resolution as soon as one becomes available.

Customers with a currently active session should continue to receive full internet access as long as they don't disconnect

Dialup Access

Fri Nov 5 09:02:15 2010

We are currently experiencing an issue with inbound call routing for Dialup/56k access. Engineers are working to resolve the problem with our telco supplier.

Wed Jul 21 14:45:02 2010

This afternoon the London Internet Exchange (LINX), where a large part of the UK's Internet traffic is routed, experienced a period of intermittent packet loss lasting less than ten minutes.

As a result, customers may have noticed a short period of connectivity problems - with web pages loading slower than normal or timing out.

This problem has now been resolved at LINX, and connectivity has been restored.

Unixweb-2

Wed Jul 7 15:20:43 2010

At approximately 3:00pm today (07-07-2010) our shared hosting platform Unixweb-2 stopped responding to network requests.

This has affected all customers who have websites hosted on this server.

We are currently liasing with engineers on-site to diagnose and rectify the problem.

An update on progress will be posted here at 3:30pm.

3:30pm UPDATE: Following a reboot, the server has returned to normal operations.

We apologise for any inconvenience this may have caused.

Email Server Work

Thu Jun 24 16:41:15 2010

Today between 1230 and 1730 we will be carrying out work on two of our email filestores. We do not expect this work to cause any customer disruption, however if you do have any issues with email please contact our support team directly on 0845 1451234

ADSL Connectivity

Wed Jun 23 15:58:49 2010

Customers using our BT ADSL products will currently be experiencing increased latency and small amounts of packet loss. Our engineers are looking into the issue and we will post an update once further information becomes available.

Customers using our LLU products are not affected

Email Delay

Tue Jun 8 15:24:40 2010

At 11am this morning, 8th June 2010, we started noticing a delay to some customers email which is stored on one of our filestores. This matter affects around 20% of our customers.

Our team have been investigating this since it was first noticed and are still working on resolving the matter.

It is now looking increasingly likely that we will have to work on the server directly and carry out essential maintanence out of hours. During this time email collection will not be possible. A further posting will be made if this is going to be the case.

The issue is not causing any emails to be lost or returned, it is however slowing delivery of them. Average delivery time to those customers affected will be up to three hours. In the meantime all emails will be queued until they can be sent.

We are very sorry for any issues this is causing you and assure you that we are doing everything to resolve this matter as quickly as possible.

If you have any further questions relating to this matter please contact your account manager or our support team on 0845 1451234

Tue Jun 8 19:41:11 2010

All domains stored on the problem filestore have been transferred to another physical machine. It is likely that customers affected by this problem will see mail deleted within the last couple of weeks reappear in their mailboxes. We apologise for this hiccup which came about while our engineers performed work to ensure that all outstanding deliveries were made.

Email issues of 18th May 2010

Wed May 19 12:58:52 2010

Dear Customers,

Some of you will be aware of the issue regarding email yesterday that affected approximately 20% of our customers, and as promised I have detailed the cause, the issues and ultimately the outcome of what happened and how it will have affected you.

On Monday evening we had a failure of part of a raid array on one of our email file-stores, where emails remain until either collected by PoP or viewed via IMAP or web mail. A failure on a hard drive is not a particular uncommon event, and scheduled emergency maintenance of the affected file-store in the early hours of Tuesday.

Unfortunately when the engineer went on site and installed the new drive it caused the file-store to break and delete the whole file-store array.

Fortunately we perform a nightly backup of customer e-mail (although we do not advertise that we do it), to try and make sure that any hardware outage will not cause irreparable harm to customers who use IMAP.

Because we had this back-up we were able to start re-building the file-store, however this did mean that we had to suspend the collection and deliver of email to those customers and domains that resided on the file-store for the duration of the rebuild.

As the day progressed we realised that the re-build was going to take substantially longer than we initially estimated and so put in a contingency plan to deliver the email temporarily from another file-store. This was implemented around 1.30 yesterday afternoon and queued email was delivered and customers had access again to the system.

Putting this contingency plan in place did however mean that we now had the affected domains with information on two file-stores, with some email (prior to the file-store crash) on each, and it meant that we would have to merge the two overnight.

Following Monday nights hardware fault, we were able to provide the e-mails up to the last backup on Sunday night. This is why on Tuesday; you were unable to access mail dated Monday.

Unfortunately last night whilst trying to restore Monday's mail from a saved drive of the original broken file-store, our engineer did not anticipate the way the backup software would behave.

As a result, the restoration caused all mail delivered yesterday (Tuesday) to be deleted.

Again, the only customers affected by this would be those using IMAP or webmail only.

Customer will now have noticed that the e-mails which were unavailable on Monday are now there, but will have instead lost Tuesday's e-mail.

And now to make things even more complicated and far less straight forward.

At the same time as the failed restoration was taking place, our engineers were busy with the continued rebuilding of the replacement of the previously broken file-store. Part of this rebuild included the re-synchronisation of the e-mail on the file-store server that is now in use. Thanks to this we now expect to be able to restore some of the e-mail but unfortunately cannot be sure when this will be complete. The process is underway as I write this, and may well be finished by the time you are reading this.

The net result of all of this is that some customers will get all yesterday's emails, when some others may only get some of them or even none at all.

As you know we pride ourselves in our support and our system up-time, and whilst this did not affect all customers, it is still a mistake that we will do our best to ensure never happens again.

You have our sincerest apologies for any disruption that was caused as a result of this.

We are human, we do make mistakes and we learn from them.

Sincerely,

Mark Cowgill,

Managing Director,

Exa Networks.

Mail Filestore Hardware Fault

Tue May 18 04:57:29 2010

Following the loss of a harddrive in one of our mail filestores on Sunday evening, we planned emergency maintenance last night to replace the faulty hardware and restore data redundancy. Unfortunately, our engineer discovered on reaching the datacenter that he was unable to rebuild the RAID array and the affected machine is currently unavailable while we rebuild it from our backups. Many customers will be unable to access mail though POP and IMAP until we are able to complete this work.

Any customer affected by the outage should be aware that our backups are up to 24 hours old and any messages you may have synced to your mail client in the past day should be safeguarded before attempting to connect to our servers once the system restoration is complete. Our support staff can be contacted if you have any further questions

Tue May 18 11:31:09 2010

Service is expected to be restored by 13:30 today. We will post further news as it becomes available

Tue May 18 13:50:00 2010

Email service is now restored. Customers will notice that mails delivered between yesterday's backup and the server crash are not currently available. Our engineers our now working to rectify this problem.