Please check out our PDF spec sheet, available for download here.

Details here

Details available on The Register.

You can see the effect of t he outage on LINX’s traffic graphs for the period …

We’re now starting to move services onto our new load-balanced platform. In particular, for WordPress sites using SuperCache you can expect to see a x10 speed increase in terms of page delivery. (although you won’t see this directly as your bandwidth / browser will be unable to keep up)

Benchmarks are showing that the new system can deliver WordPress based web pages at a rate of 20M bytes / second.

Here’s the feedback from the Engineers;

Around 11:16pm our primary power supply failed in our Cisco core, and shortly after it’s second PSU also blew, taking out the breaker on the comms cabinet in room one. We tried to repair the PSU at circuit level but alas, were unsuccessful and declared it a lost cause at approx 1am. We tried to introduce our foundry router to the THN links but a fault light was on the BT Openreach WES box, so we called the BT NOC at around 1:45am to report a fault on the circuits. We were replied to at around 2:15am with “no fault found” , “customer equipment faulty” – Following this we spent the next few hours attempting a workaround with three different switches, LH/LX optics etc. – All with the same result.

At 8am, Hardware.com despatched two replacement PSU’s by dedicated delivery driver, these only arrived just after 11am. The switch was up and running, and as logs show, customer racks were reconnected at around 11:15am, but our main fibre and redundant pair were both*still* showing faults on the NTE unit. BT Openreach were called again at around 12 noon, and we demanded site presence. The engineer arrived onsite first at 3pm, then 4pm, having completed “end to end loopback” tests on both fibre pairs – No fault found.  I then took it upon myself to close the front gate and take hostage the BT Openreach engineer, who was determined he was finishing his shift, and would return the following day – Naturally, this wasn’t going to happen. After quizzing him repeatedly regarding whether his colleague at Star in Barnwood, had ruled out the line card or done a reboot, I finally convinced him to pull the card at the remote end for a cold reboot.

At around 5:30pm the first fibre pair became active, green lights, and we began exchanging packets of data with Telehouse North once again.

Many thanks to the guys at the Data Center who stuck with the fault from 11:16pm last night until 5:45pm today when BT managed to get the circuits back on-line. Time for a well-earned rest. Following on, this is the work currently underway to mitigate the effects any such future problems;

We have placed on order, two Cisco 6509-E chassis, two Sup 720 3B cards, and a plethora of Cisco line cards, with spare optics etc. Shortly we’ll have a pair of fully redundant cores, powered from a pair of separate Riello UPS systems, A + B fed power supplies in each unit. Our Gas Turbines and new 11Kv feed are already onsite and ready for commissioning at the beginning of next month, and we’ll be posting various photographs of the hardware, power circuitry, and our new suite.

The Date Center’s website (Saxon Data) can be found here.

Post Mortem from the Data Center Team (May 7)

Saxon Data Incident Report
Date of Incident : 2010-04-19
Incident Type : Communications Loss

What happened

At 23:30 (BST) on Monday, 19th April 2010, we experienced a multiple power supply failure to our Cisco 6509 router directly impacting connectivity to Saxon House. Replacement PSU’s were immediately sourced from our hardware vendor but were unavailable until the following morning.

At 10:24, the replacement PSU’s arrived and were fitted by on-site engineers. The router booted successfully but failed to establish a layer 1 connection to our Telehouse router located in London, Docklands. Upon further investigation, the BT’s remote NTE which carries the first leg of layer 1 fibre connectivity to London, had appeared to reset during the outage.

A BT engineer was dispatched to the local Barnwood exchange to diagnose and correct the outstanding fault. At 17:30, full connectivity was restored and services resumed.

What we learned

1. Redundant power supplies are not always enough to ensure that a router stays up.
2. Our fallback procedure doesn’t allow us to communicate with all of our customers in a timely manner.
3. Multiple, coincidental failures are more common than we appreciate.

What we are doing to improve

Obviously, our customers cannot tolerate this length of outage and we must make sure that we minimalise the risk of anything similar happening again. To that end, we are bringing forward the acquisition of another Cisco router which should be installed by the time this weekend is out. For the technical people out there, we will be running our new cores on the following rig:

Router 1
Cisco 6509 Chassis
2 x 2500W Power Suplies
1 x High speed fan tray (plus 1 spare for each router)
1 x WS-SUP720-3B-GR3 – Supervisor fabric (handling main inbound dual fibre feeds)
1 x WS-X6748-GE-TX-GR3 – 48 port Gigabit line cards for customer racks
1 x WS-X6724-SFP-GR3 – 24 Port SFP fibre card for customer racks

Router 2
Cisco 6509 Chassis
2 x 1300W Power supplies
1 x High speed fan tray (plus 1 spare for each router)
1 x WS-SUP720-GR3
1 x 16 port classic fibre card
1 x 48 port Ethernet card

We’ll be running dual OSPF sessions (1 per router) to the KCN rack in Telehouse North, over the existing BT NTE equipment and also the second, redundant fibre. This will mean if any part of a router fails, we have a spare onsite and if a complete catastrophic failure occurs, it’s a matter of several minutes work by hands to swap everyone over to the spare box.

We’re currently holding over 150 pieces of spare SFP optics – SX and LH/LX, so a trivial failure of one single port is catered for also.

Each router will be power from independent N+1 UPS feeds, and this will also be an available option to rack customers at some time real soon.

In addition, we have a 2 hour service contract arranged with Hardware.com who are based in Cirencester – 20 minutes from us.

We’re also bringing forward investment in our Power generation and UPS infrastructure to maintain a high degree of redundancy. As an example, we have taken delivery of three Riello master plus units to be run in parallel, and have another one on the way, which will bring us up to a total 400kva 2N.

All units have output transformers, and input rectifiers, 8 minutes battery autonomy at full load. We have arranged to light the second fibre in the BT OpenReach blowpipe, and to have it bypass the NTE equipment at both ends of the circuit, ensuring that a failing NTE can no longer cause failure of the data path.

We are working to ensure that an off-site copy of the ticketing system will be available in the future in the event that the data centre suffers a communications outage.

As we expand we will install a second, diverse fibre feed to avoid further downtime due to events such as fibre damage during civil engineering work. We have meetings with two major communications providers in the next month, both of whom are eager to discuss POPing our building with their own Dark Fibre feeds. Naturally, this would be a massive advantage to both the facility and its customers, so we’ll keep you all updated.

Timescale / router upgrade plans

On Thursday 6th we are looking to receive the last of the parts for the new router and will have fresh Multimode cable into each rack. The new router will at first be placed in line with the old one and all customer VLANs will continue to be fed from the old system. Customers will be notified of the exact time of this emergency changeover. As settings will be done beforehand, this should result in an outage of less than 10 seconds, whilst the fibres are swapped over.

We’ll let everyone know once this phase is complete and invite service tickets, where you will get the chance to specify the exact time of day we switch you over to the new core. We plan on leaving the old fibres into each rack for extra redundancy and to save time should a cabling fault ever affect you.

Once everyone is over to the new router, we’ll be looking to pull the old 6509 chassis and replace with the new spare unit / Sup card / PSU’s etc. And setup the second fibre link.

Naturally, we’ll keep everyone informed throughout the whole process by email and if anyone has any questions, please feel free to call / email.

Please accept our most sincere and deepest apologies and feel free to copy this document to any of your clients who may be asking questions of you. You can all rest assured that the process of upgrading connectivity, power and UPS / generation is completely underway and will result in a highly resilient DC infrastructure in a very short while.

Thanks
Saxon Data technical support team.

Here’s a quick-start guide to Web programming with ExtJS, one of the many options out there if you want to write modern online applications. (For a taster of what can be achieved, please take a look at the ExtJS demos)

What you will need

First you’ll want a web server with PHP installed, typically using your local desktop machine will be quite sufficient and indeed all the components required are included in most standard Linux distributions. Also, I’m going to show you how to extract information from an SQL database, so you’ll want to install a copy of MySQL and the PHP/MySQL interface.

Setting up

Next you’ll want a copy of ExtJS, so down load either version’s 3.1 or 3.2 from ExtJS.com and unpack them into the root folder of your webserver, on Linux this is usually /var/www.  For convenience, copy the contents (including sub-folders) of extjs/examples/direct/php into /var/www/remote, this folder contains the files you’ll need to allow your browser based application to talk to your server.

Now for some html

Create a file in /var/www called index.html and paste in the following;

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>EXTJS Example</title>
<link rel="stylesheet" type="text/css" href="extjs/resources/css/ext-all.css" />
<script type="text/javascript" src="extjs/adapter/ext/ext-base.js"></script>
<script type="text/javascript" src="extjs/ext-all-debug.js"></script>
<script type="text/javascript" src="remote/api.php"></script>
<script type="text/javascript" src="example.js"></script>
</head>
<body>
</body>
</html>

A bit of JavaScript

Next we’re going to load some javascript into example.js, this should also sit in /var/www and is referenced by the html file above. The ‘viewport’ will render to the page or document body which was defined in the html file above as being empty.

Ext.Direct.addProvider(Ext.app.REMOTING_API);

Ext.onReady(function(){
   var panel = new Ext.Viewport({
      layout   : 'fit',
        defaults : {
         border     : true,
         frame      : true
      },
      items : [grid]
   });
});

Note that we’ve not yet defined grid, so this code needs also to come inside of the onReady function, but before the definition of the panel variable.

var grid = new Ext.grid.GridPanel({
   store: store,
   columns: [
      { width: 100,   header: 'IP Address', dataIndex: 'ip'     },
      { width: 100,   header: 'Host Name',  dataIndex: 'name'   },
      { width: 100,   header: 'Status',     dataIndex: 'status' }
   ]
});

As a little explanation; store is something else we’ve not yet defined. The the remainder of the code defines a table with columns “IP Address”, “Host Name” and “Status”. Next we need to define store which is the client side interface to the appropriate MySQL table, or to be more specific, MySQL query.

store = function()
{
   return new Ext.data.DirectStore({
      directFn : direct.hostList,
      autoLoad : true,
      fields   : [
         { name: 'ip'     , type: 'string' },
         { name: 'name'   , type: 'string' },
         { name: 'status' , type: 'number' }
      ]
   });
}

As far as code that’s going to run ‘in’ your browser is concerned, we should be about done for the moment and the next snippets are code for the server side of the equation. First we start by editing config.php in your remote folder (this is the folder you created / copied above). You need to make it look like this;

<?php
$API = array(
   'direct'=>array(
      'methods'=>array(
         'hostList'=>array( 'len'=>1 )
      )
   )
);

This is telling the ExtJS code how to locate the correct methods to call when your store components asks for it’s table to be populated with data. Next comes the scary PHP code, note that when you get to grips with this it’s really is pretty trivial.

The following needs to be called direct.php and goes into the direct/classes folder.

<?php

function report($code,$text) {
   return array( array('status'=>$code,'text'=>$text));
}

define('DB_NAME'      , '<database>');
define('DB_USER'      , '<user>');
define('DB_PASSWORD'  , '<password>');
define('DB_HOST'      , '<host>');

class direct {

   function hostList($data)
   {
      $db = mysql_connect(DB_HOST,DB_USER,DB_PASSWORD,true);
      if(!db) return report(0,mysql_error());
      if(!mysql_select_db(DB_NAME)) return report(0,'Unable to select database');
      $sql = 'SELECT * FROM hosts';
      $results = mysql_query($sql,$db);
      if(!$results) return report(0,mysql_error());
      $out = array();
      while( $row = mysql_fetch_assoc($results) )
      {
         array_push($out,array('ip'=>$row['ip'],'name'=>$row['hostname'],'status'=>$row['response']);
      }
      return $out;
   }
}

For this to work you will obviously need to have a database with matching name and login credentials as specified in the four define lines. If you want to re-create this MySQL table, you will need something like;

CREATE TABLE `hosts` (
   `id` smallint(6) NOT NULL AUTO_INCREMENT,
   `ip` varchar(15) DEFAULT NULL,
   `hostname` varchar(32) DEFAULT NULL,
   `response` varchar(15) DEFAULT NULL,
   PRIMARY KEY (`id`)
) ENGINE=MyISAM;

What does it look like

It may not be much to look at, but if you consider what you’re doing in the context of the framework, you already have something which can form the basis of a fairly complex application.

To see a ‘complete’ online version in action, together with an ‘assembled’ version of example.js, please visit this link.

Where to go from here?

First you should add some error trapping, the roots of which are already inserted into direct.php, then you could try looking at passing parameters through to the store for filtered loads, and calling ‘Ext.direct’ ‘directly’ without going through a store (which is very useful for obtaining small amount of information or simply performing updates).

Expect to see some ‘complete’ projects appear here over coming months ..