Details available on The Register.

You can see the effect of t he outage on LINX’s traffic graphs for the period …

We’re now starting to move services onto our new load-balanced platform. In particular, for Wordpress sites using SuperCache you can expect to see a x10 speed increase in terms of page delivery. (although you won’t see this directly as your bandwidth / browser will be unable to keep up)

Benchmarks are showing that the new system can deliver Wordpress based web pages at a rate of 20M bytes / second.

Here’s the feedback from the Engineers;

Around 11:16pm our primary power supply failed in our Cisco core, and shortly after it’s second PSU also blew, taking out the breaker on the comms cabinet in room one. We tried to repair the PSU at circuit level but alas, were unsuccessful and declared it a lost cause at approx 1am. We tried to introduce our foundry router to the THN links but a fault light was on the BT Openreach WES box, so we called the BT NOC at around 1:45am to report a fault on the circuits. We were replied to at around 2:15am with “no fault found” , “customer equipment faulty” – Following this we spent the next few hours attempting a workaround with three different switches, LH/LX optics etc. – All with the same result.

At 8am, Hardware.com despatched two replacement PSU’s by dedicated delivery driver, these only arrived just after 11am. The switch was up and running, and as logs show, customer racks were reconnected at around 11:15am, but our main fibre and redundant pair were both*still* showing faults on the NTE unit. BT Openreach were called again at around 12 noon, and we demanded site presence. The engineer arrived onsite first at 3pm, then 4pm, having completed “end to end loopback” tests on both fibre pairs – No fault found.  I then took it upon myself to close the front gate and take hostage the BT Openreach engineer, who was determined he was finishing his shift, and would return the following day – Naturally, this wasn’t going to happen. After quizzing him repeatedly regarding whether his colleague at Star in Barnwood, had ruled out the line card or done a reboot, I finally convinced him to pull the card at the remote end for a cold reboot.

At around 5:30pm the first fibre pair became active, green lights, and we began exchanging packets of data with Telehouse North once again.

Many thanks to the guys at the Data Center who stuck with the fault from 11:16pm last night until 5:45pm today when BT managed to get the circuits back on-line. Time for a well-earned rest. Following on, this is the work currently underway to mitigate the effects any such future problems;

We have placed on order, two Cisco 6509-E chassis, two Sup 720 3B cards, and a plethora of Cisco line cards, with spare optics etc. Shortly we’ll have a pair of fully redundant cores, powered from a pair of separate Riello UPS systems, A + B fed power supplies in each unit. Our Gas Turbines and new 11Kv feed are already onsite and ready for commissioning at the beginning of next month, and we’ll be posting various photographs of the hardware, power circuitry, and our new suite.

The Date Center’s website (Saxon Data) can be found here.

Post Mortem from the Data Center Team (May 7)

Saxon Data Incident Report
Date of Incident : 2010-04-19
Incident Type : Communications Loss

What happened

At 23:30 (BST) on Monday, 19th April 2010, we experienced a multiple power supply failure to our Cisco 6509 router directly impacting connectivity to Saxon House. Replacement PSU’s were immediately sourced from our hardware vendor but were unavailable until the following morning.

At 10:24, the replacement PSU’s arrived and were fitted by on-site engineers. The router booted successfully but failed to establish a layer 1 connection to our Telehouse router located in London, Docklands. Upon further investigation, the BT’s remote NTE which carries the first leg of layer 1 fibre connectivity to London, had appeared to reset during the outage.

A BT engineer was dispatched to the local Barnwood exchange to diagnose and correct the outstanding fault. At 17:30, full connectivity was restored and services resumed.

What we learned

1. Redundant power supplies are not always enough to ensure that a router stays up.
2. Our fallback procedure doesn’t allow us to communicate with all of our customers in a timely manner.
3. Multiple, coincidental failures are more common than we appreciate.

What we are doing to improve

Obviously, our customers cannot tolerate this length of outage and we must make sure that we minimalise the risk of anything similar happening again. To that end, we are bringing forward the acquisition of another Cisco router which should be installed by the time this weekend is out. For the technical people out there, we will be running our new cores on the following rig:

Router 1
Cisco 6509 Chassis
2 x 2500W Power Suplies
1 x High speed fan tray (plus 1 spare for each router)
1 x WS-SUP720-3B-GR3 – Supervisor fabric (handling main inbound dual fibre feeds)
1 x WS-X6748-GE-TX-GR3 – 48 port Gigabit line cards for customer racks
1 x WS-X6724-SFP-GR3 – 24 Port SFP fibre card for customer racks

Router 2
Cisco 6509 Chassis
2 x 1300W Power supplies
1 x High speed fan tray (plus 1 spare for each router)
1 x WS-SUP720-GR3
1 x 16 port classic fibre card
1 x 48 port Ethernet card

We’ll be running dual OSPF sessions (1 per router) to the KCN rack in Telehouse North, over the existing BT NTE equipment and also the second, redundant fibre. This will mean if any part of a router fails, we have a spare onsite and if a complete catastrophic failure occurs, it’s a matter of several minutes work by hands to swap everyone over to the spare box.

We’re currently holding over 150 pieces of spare SFP optics – SX and LH/LX, so a trivial failure of one single port is catered for also.

Each router will be power from independent N+1 UPS feeds, and this will also be an available option to rack customers at some time real soon.

In addition, we have a 2 hour service contract arranged with Hardware.com who are based in Cirencester – 20 minutes from us.

We’re also bringing forward investment in our Power generation and UPS infrastructure to maintain a high degree of redundancy. As an example, we have taken delivery of three Riello master plus units to be run in parallel, and have another one on the way, which will bring us up to a total 400kva 2N.

All units have output transformers, and input rectifiers, 8 minutes battery autonomy at full load. We have arranged to light the second fibre in the BT OpenReach blowpipe, and to have it bypass the NTE equipment at both ends of the circuit, ensuring that a failing NTE can no longer cause failure of the data path.

We are working to ensure that an off-site copy of the ticketing system will be available in the future in the event that the data centre suffers a communications outage.

As we expand we will install a second, diverse fibre feed to avoid further downtime due to events such as fibre damage during civil engineering work. We have meetings with two major communications providers in the next month, both of whom are eager to discuss POPing our building with their own Dark Fibre feeds. Naturally, this would be a massive advantage to both the facility and its customers, so we’ll keep you all updated.

Timescale / router upgrade plans

On Thursday 6th we are looking to receive the last of the parts for the new router and will have fresh Multimode cable into each rack. The new router will at first be placed in line with the old one and all customer VLANs will continue to be fed from the old system. Customers will be notified of the exact time of this emergency changeover. As settings will be done beforehand, this should result in an outage of less than 10 seconds, whilst the fibres are swapped over.

We’ll let everyone know once this phase is complete and invite service tickets, where you will get the chance to specify the exact time of day we switch you over to the new core. We plan on leaving the old fibres into each rack for extra redundancy and to save time should a cabling fault ever affect you.

Once everyone is over to the new router, we’ll be looking to pull the old 6509 chassis and replace with the new spare unit / Sup card / PSU’s etc. And setup the second fibre link.

Naturally, we’ll keep everyone informed throughout the whole process by email and if anyone has any questions, please feel free to call / email.

Please accept our most sincere and deepest apologies and feel free to copy this document to any of your clients who may be asking questions of you. You can all rest assured that the process of upgrading connectivity, power and UPS / generation is completely underway and will result in a highly resilient DC infrastructure in a very short while.

Thanks
Saxon Data technical support team.

Here’s a quick-start guide to Web programming with ExtJS, one of the many options out there if you want to write modern online applications. (For a taster of what can be achieved, please take a look at the ExtJS demos)

What you will need

First you’ll want a web server with PHP installed, typically using your local desktop machine will be quite sufficient and indeed all the components required are included in most standard Linux distributions. Also, I’m going to show you how to extract information from an SQL database, so you’ll want to install a copy of MySQL and the PHP/MySQL interface.

Setting up

Next you’ll want a copy of ExtJS, so down load either version’s 3.1 or 3.2 from ExtJS.com and unpack them into the root folder of your webserver, on Linux this is usually /var/www.  For convenience, copy the contents (including sub-folders) of extjs/examples/direct/php into /var/www/remote, this folder contains the files you’ll need to allow your browser based application to talk to your server.

Now for some html

Create a file in /var/www called index.html and paste in the following;

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>EXTJS Example</title>
<link rel="stylesheet" type="text/css" href="extjs/resources/css/ext-all.css" />
<script type="text/javascript" src="extjs/adapter/ext/ext-base.js"></script>
<script type="text/javascript" src="extjs/ext-all-debug.js"></script>
<script type="text/javascript" src="remote/api.php"></script>
<script type="text/javascript" src="example.js"></script>
</head>
<body>
</body>
</html>

A bit of JavaScript

Next we’re going to load some javascript into example.js, this should also sit in /var/www and is referenced by the html file above. The ‘viewport’ will render to the page or document body which was defined in the html file above as being empty.

Ext.Direct.addProvider(Ext.app.REMOTING_API);

Ext.onReady(function(){
   var panel = new Ext.Viewport({
      layout   : 'fit',
        defaults : {
         border     : true,
         frame      : true
      },
      items : [grid]
   });
});

Note that we’ve not yet defined grid, so this code needs also to come inside of the onReady function, but before the definition of the panel variable.

var grid = new Ext.grid.GridPanel({
   store: store,
   columns: [
      { width: 100,   header: 'IP Address', dataIndex: 'ip'     },
      { width: 100,   header: 'Host Name',  dataIndex: 'name'   },
      { width: 100,   header: 'Status',     dataIndex: 'status' }
   ]
});

As a little explanation; store is something else we’ve not yet defined. The the remainder of the code defines a table with columns “IP Address”, “Host Name” and “Status”. Next we need to define store which is the client side interface to the appropriate MySQL table, or to be more specific, MySQL query.

store = function()
{
   return new Ext.data.DirectStore({
      directFn : direct.hostList,
      autoLoad : true,
      fields   : [
         { name: 'ip'     , type: 'string' },
         { name: 'name'   , type: 'string' },
         { name: 'status' , type: 'number' }
      ]
   });
}

As far as code that’s going to run ‘in’ your browser is concerned, we should be about done for the moment and the next snippets are code for the server side of the equation. First we start by editing config.php in your remote folder (this is the folder you created / copied above). You need to make it look like this;

<?php
$API = array(
   'direct'=>array(
      'methods'=>array(
         'hostList'=>array( 'len'=>1 )
      )
   )
);

This is telling the ExtJS code how to locate the correct methods to call when your store components asks for it’s table to be populated with data. Next comes the scary PHP code, note that when you get to grips with this it’s really is pretty trivial.

The following needs to be called direct.php and goes into the direct/classes folder.

<?php

function report($code,$text) {
   return array( array('status'=>$code,'text'=>$text));
}

define('DB_NAME'      , '<database>');
define('DB_USER'      , '<user>');
define('DB_PASSWORD'  , '<password>');
define('DB_HOST'      , '<host>');

class direct {

   function hostList($data)
   {
      $db = mysql_connect(DB_HOST,DB_USER,DB_PASSWORD,true);
      if(!db) return report(0,mysql_error());
      if(!mysql_select_db(DB_NAME)) return report(0,'Unable to select database');
      $sql = 'SELECT * FROM hosts';
      $results = mysql_query($sql,$db);
      if(!$results) return report(0,mysql_error());
      $out = array();
      while( $row = mysql_fetch_assoc($results) )
      {
         array_push($out,array('ip'=>$row['ip'],'name'=>$row['hostname'],'status'=>$row['response']);
      }
      return $out;
   }
}

For this to work you will obviously need to have a database with matching name and login credentials as specified in the four define lines. If you want to re-create this MySQL table, you will need something like;

CREATE TABLE `hosts` (
   `id` smallint(6) NOT NULL AUTO_INCREMENT,
   `ip` varchar(15) DEFAULT NULL,
   `hostname` varchar(32) DEFAULT NULL,
   `response` varchar(15) DEFAULT NULL,
   PRIMARY KEY (`id`)
) ENGINE=MyISAM;

What does it look like

It may not be much to look at, but if you consider what you’re doing in the context of the framework, you already have something which can form the basis of a fairly complex application.

To see a ‘complete’ online version in action, together with an ‘assembled’ version of example.js, please visit this link.

Where to go from here?

First you should add some error trapping, the roots of which are already inserted into direct.php, then you could try looking at passing parameters through to the store for filtered loads, and calling ‘Ext.direct’ ‘directly’ without going through a store (which is very useful for obtaining small amount of information or simply performing updates).

Expect to see some ‘complete’ projects appear here over coming months ..

What does this mean in real terms?

• It’s very quick
• It can scale up to hundred’s of machines if necessary
• It supports XCACHE APC and Memcached for PHP acceleration
• Each customer runs within their own UID space, so it’s secure
• Multiple mirrored  SQL servers are available for your applications
• We can provide these features where other’s can’t at a fraction of the price you would pay for equivalent dedicated setups

What does it look like?

Our first site is up at http://forum.linux.co.uk, take a look and tell us what you think!

Here’s a summary of SMF’s features;

• Uses PHP and MySQL.
• Comprehensive template system.
• Advanced permission and user management.
• Supports multiple languages at once.
• Open and well-documented source code.
• Tracking of new and old unread topics, not just from your last visit.
• Designed for optimal performance and scalability.
• Multi-media output. (XHTML, XML, RSS, WAP)
• Multi-language support from a large community.
• Package manager that automatically installs or uninstalls mods (also known as hacks.)
• Search:
  • Search the entire forum, a category/board or within a topic.
  • Search within your personal messages.

• All actions seamlessly require a session based authorization code.
• Administrative actions require the user’s password (and do not rely solely on cookies.)
• Major actions are time and IP locked, preventing ‘hammering’.
• The number of login attempts from a certain IP can be limited and time locked.

• Ability to display page creation time and query count per page.
• Put a board into maintenance mode, allowing only admins to login.
• Word censoring, either full word or partial.
• Ability to break up long words.
• Group boards into collapsible categories.
• Set categories as non-collapsible.
• Reorder boards within categories, or reorder categories.
• Create child boards under other boards. (sub boards)
• Assign moderators to boards.
• Allow certain membergroups to access a board, including guests only.
• Configure permissions for each membergroup on the board level.
• Ability to indicate new posts to child boards but nothing new in parent.
• Require registration before forum entrance.
• Require a user to agree to terms before they register.
• Disable member registration completely (allowing only moderators to register people).
• Require email authentication by sending an authentication link.
• Require a moderator to approve registration.
• Register new members from the admin center.
• Several security checks during navigation.
• Password reminder option, by email with confirmation. (doesn’t automatically reset your password.)
• Both cookie and session based authentication (works without cookies.)
• Cookies can be set local to a path, global to all subdomains, or normally.
• Adjustable expiration time for authentication cookies.
• Sortable and searchable public memberlist (accessible by a permission.)
• Powerful sortable and searchable admin memberlist.
• Show all (error) messages and IPs made by a member. (track user)
• Show all (error) messages from an IP address or range. (track IP)
• See who’s doing what (accessible by permission.)
• Several board statistics (accessible by permission.)
• Tracking of member’s online time in seconds.
• Tracking of topics, messages, new members, and hits per day.
• Individual member statistics accessible from their profile.
• Ability to create announcement boards (members receive a notification of topics automatically.)
• Member option to disable receiving announcements.
• Email or private message your members by membergroup.
• Show a news ticker or news fader.
• Ability to choose sendmail or SMTP (with or without authentication.)
• Ability to send a topic to a friend.
• Ability to view a “printer friendly” version of topics.
• Create membergroups to group members on permissions, access rights, and/or appearance.
• Assign several membergroups to a single member, with one membergroup as the primary group.
• Define membergroups that are auto-assigned based on the amount of posts a user has.
• Determine the maximum number of personal messages a membergroup is allowed to have by group.
• Assign graphical symbol(s) to a membergroup – by primary group.
• Determine which membergroups are allowed to access a board.
• Ban members based on their username, email address, IP address or hostname.
• Support of wildcards for email address, IP address, and hostname.
• Include a ban reason (viewable for the banned user).
• Include a ban note (only viewable by the admins).
• Chose between full ban, ‘no post’ ban, or registration ban.
• Include an expiration time for bans.

• Integrate parts of forum software into existing website using SSI or PHP.
  • Recent topics or posts.
  • Recent news posts in a board.
  • Recent polls.
  • Several forum statistics.
  • List of users online.
  • The menu bar.
  • News.
  • Search.
  • Login/logout.
  • Today’s events/birthdays/holidays.
• Export forum data using XML/RSS
  • Latest members.
  • News.
  • Recent posts.

• Support for WAP, WAP2 and I-mode protocols.
• Browse through boards/topics/messages with reduced page size.
• Ability to login and logout (not with WAP 1.)
• Ability to see new topics/boards with new replies.
• Ability to jump to first unread reply.
• Ability to post new messages (not with WAP 1.)

• Ability to allow or disallow your users to select their own theme.
• Ability to reset all of your members to a certain theme.
• Ability to install a new theme via your administration center.
• Default templates are XHTML 1.0 Transitional and CSS 2.0 compliant.
• Admin can add smileys and smiley sets.
• Members can choose which smiley set they wish to use (or none.)
• Themes can be installed by way of the “latest and greatest themes” panel.
• SSI can have and show layers and the like from the template system.

• Spell Check.
• Quick Reply
  • Members can disable it or collapse it.
  • Can be used with “Quote”.
  • Also contains “Spell Check”.
• Vast number of “bulletin board codes” to use (including rtl, acronym, and others.)
• Optional editing grace period before a post is shown as modified.
• ‘Insert Quote’ feature on posting screen to quickly quote previous replies.

• Can be added or removed to existing topics.
• Ability to set expiration date.
• Ability to hide results till expiration of poll.
• Ability to hide results until after people have voted.
• Ability to determine how many votes a user may cast.
• Polls can have up to 256 options.

• Allowed on a per member group or board basis.
• The filename can be encrypted to increase the security (so you can upload .php files, etc.)
• Optional restriction on what filetypes may be uploaded.
• Ability to restrict such that only registered members can view them (disables hotlinking, mostly.)
• Images can be embedded into posts.

• Shows Birthdays, Events and Holidays.
• Display calendar information on board index.
• Link an existing topic to the calendar.
• Allow only certain members/groups post events.
• Event poster can edit event.
• Events can span multiple days.

• Ability to lock and unlock a topic.
  • Topics which are locked by an administrator can not unlocked by anyone else.
• Ability to allow members to use moderation on just topics they posted.
• “Report to moderator” link.
• Delete or modify posts.
• Ability to Merge/Split topics.
• Move or delete topics.
• Sticky or unsticky topics.
• Recycle bin/board for all deleted topics and posts.
• Moderation log which shows moderation activity and disallows log removal outside of 1 day.
• AJAX editing:
  • Edit a post without page reloads.
  • Double click to edit topic subjects on message index.

• Ability to hide your email address from the public.
• Select their favorite theme among the activated forum themes.
• Select their smiley set from the ones available.
• Email notification
  • New topics per board.
  • Single topics.
  • Private messages.
• Login with ‘invisible’ mode.
• Options to hide other members’ avatars and signatures.
• Ability to download and resize uploaded avatars.
• Personal text, avatar, signature, and all the niceties of forums can be used and are kept up to date in all your posts.

• Icons for topics you’ve posted to.
• Show new replies to your posts.
• Collapsible ‘Additional Options’ on post screen.
• Ability to view most recent messages.
• List all topics that have new replies since your last visit.
• See who is browsing the same boards or topics as you.
• Topics that span multiple pages have the page numbers listed as well as an “all” link.

The Timeline widget itself is fairly ‘raw’ so we’ve coded it up to automatically build the timeline from an SQL table, then written a native program to edit the data. It’s relatively trivial, however it does run over the net so it means that a number of people can update the timeline simultaneously fro a number of remote locations.

Changes to the database trigger a script which automatically builds a new static XML file for optimal online viewing performance.

Logging in ...

Browsing / Editing the timeline ...

One of the main features of the new site (apart from it being based on Wordpress rather than Plone) is it’s ability to run blogs and integrate with to external systems like Twitter. So, with any luck news updates should be far more frequent and comprehensive than on the old system and you have the option to comment on news items if you so wish.