1 Thursday, 12 January 2023 2 (1.00 pm) 3 MR BLAKE: Good afternoon, sir. 4 SIR WYN WILLIAMS: Good afternoon. 5 MR BLAKE: Can I call Mr Muchow. 6 SIR WYN WILLIAMS: Certainly. 7 STEPHEN MUCHOW (affirmed) 8 Questioned by MR BLAKE 9 SIR WYN WILLIAMS: Good afternoon, Mr Muchow. I hope you 10 haven't had too difficult a time in getting to the 11 Inquiry this afternoon. 12 A. You obviously have heard. Yes, my train was cancelled 13 and then my seat was taken on the next one. 14 SIR WYN WILLIAMS: If it's of any consolation to you, if 15 I had been attempting to come, I wouldn't even have got 16 out of the town in which I live. So at least you got 17 there. 18 A. I texted my wife to say much the same for the future. 19 MR BLAKE: Thank you. Can you give your full name, please? 20 A. Stephen Manfred Muchow. 21 Q. Mr Muchow, do you have in front of you a witness 22 statement? 23 A. I do. 24 Q. Is that statement dated 12 September of this year? 25 A. It is. 1 1 Q. Could I ask you to have a look at the final page, 2 page 22 of 24. 3 A. Yes. That's my signature. 4 Q. Thank you. Is that statement true to the best of your 5 knowledge and belief? 6 A. Yes, it is. 7 Q. Thank you very much, Mr Muchow. That statement is going 8 to go into evidence and it will be uploaded onto the 9 Inquiry's website so the questions I'm going to ask you 10 today will be in addition to the questions you have 11 already been asked about in that statement. But I'm 12 going to start with a bit of background. You joined ICL 13 in 1979; is that correct? 14 A. Yes. 15 Q. You held various roles in Pathway and then Fujitsu until 16 your retirement in 2009? 17 A. I did, yes. 18 Q. Much of your time at Pathway was in the customer 19 services division; is that right? 20 A. Yes. 21 Q. I think between 1985 and 2001 you were in that division? 22 A. In fact, most of my career in ICL has been with customer 23 service. 24 Q. And you were customer services director at the time of 25 the rollout of Horizon? 2 1 A. No, I was customer service director, yes. I beg your 2 pardon. 3 Q. I think in 2001 you became business director? 4 A. Yes, there were some organisational changes in the 5 offing and I became interim managing director, which is 6 business director. 7 Q. So for the core period that we're going to be addressing 8 today you were customer services director? 9 A. Yes. 10 Q. I'd like to start by looking at the hierarchy of ICL at 11 the time. Can we look at POL00028211, please. Thank 12 you. This is part of the codified agreement. If we 13 turn over the page, it has -- could we look at the 14 structure there, the diagram at the bottom of the screen 15 there. Thank you very much. 16 So we have there your name, it's the second one 17 from the right-hand side, director customer services; is 18 that right? 19 A. Yes. 20 Q. There's quite a flat structure. Can you just explain to 21 us how that worked with the various directors and who 22 they reported to. 23 A. Well, we all reported to John Bennett. This was 24 a contract, a PFI contract initially, and there were 25 myriad streams of work and expertise required and so all 3 1 of these people here were responsible for a very 2 specific part of the bid for the contract and 3 subsequently some of us remained on to operate. 4 Q. So in terms of your reporting line, is it Mr Coombs and 5 Mr Bennett or principally Mr Bennett or principally 6 Mr Coombs? 7 A. Principally Mr Bennett but my recollection of Mike 8 Coombs was that he was a very difficult man to ignore 9 and he had a great sway. 10 Q. Was it straightforward to report concerns to the 11 managing director and the deputy managing director? 12 A. Was it straightforward? 13 Q. Yes, in terms of the reporting lines and their 14 management style for example. 15 A. Yes. I think John Bennett -- he chose the people that 16 he wanted to do these roles. He interviewed us. 17 I remember there being quite a team spirit. This was 18 a very large bid that we were mounting, one of the 19 largest, I think, that ICL Fujitsu had ever done, and 20 I think it depended on a great deal of teamwork. So we 21 were very much a team and John, as I recall, was 22 somebody who -- he had an open door and we knew what we 23 had to do. He set our objectives and we got on with it. 24 Q. Can we look at page 16, which sets out your CV in a bit 25 of detail. Thank you. I'm just going to read that 4 1 section, 1996 to the present, so in 1996 to the time of 2 the contract. It describes your role as: 3 "Director Customer Service, Pathway. 4 "Responsible for all aspects of Customer Service 5 across all boundaries both internal ... and external and 6 with subcontractors." 7 A. Yes. 8 Q. If we go over the page, it sets out your role in 9 a little more depth, and it says there: 10 "Role. 11 "Operate services in accordance with service level 12 agreements ... The current role includes: 13 "Client 14 "POCL operational support services 15 "Help desks 16 "Counter support services 17 "Site services 18 "Training (ongoing) 19 "Management information ... 20 "Pathway 21 "Help desks 22 "Site services 23 "Training 24 "MIS", I think is management information? 25 A. It is. There's clearly a few typos in there. 5 1 Q. But it's fair to say your role covered both helpdesks 2 and training? 3 A. If it wasn't to do with development, then operationally 4 it was for me to deal with. So I didn't do any 5 development, I didn't do any implementation but my team 6 looked after the operation of the data centres, the 7 support services, the management information system, 8 which was actually probably one of the largest 9 components of that because we were building something 10 from scratch. We didn't have anything available 11 off-the-shelf. This all had to be built. 12 Q. What does that mean, management information services 13 very briefly? 14 A. Management information systems, not services, the 15 systems which, for instance, we had to submit every 16 month how well we'd done in achieving the service level 17 agreements that we'd signed up to do. So we had to 18 devise ways of showing how the helpdesks had responded, 19 how the transactions had performed -- I'm sure you will 20 come on to this -- how many incomplete transactions and 21 lost transactions there were, and so on. 22 And all of that was done by my team in devising 23 processes, procedures and spreadsheets and other forms, 24 that maybe there were some databases written by the SSC, 25 data applications to capture the information that we 6 1 gleaned from the system, in order to inform the 2 management team how well we were doing. 3 Q. I'm going to begin, just by way of background, to ask 4 you some questions about the Helpdesk. We have heard 5 the about Helpdesk, both in the previous phase and this 6 phase, but just to refresh our memory can you tell us, 7 in very brief terms, what the Helpdesk was, the Horizon 8 System Helpdesk how that differed from, for example, the 9 National Business Support Centre. 10 A. Yes. The Horizon System Helpdesk was the first point of 11 contact for most things that were unexplained, went 12 wrong, confused in the system. So the postmasters and 13 their staff would call the Horizon System Helpdesk when 14 something didn't go right. 15 The first line of support was the HSH where they 16 would log into a system called PowerHelp, which was 17 a global ICL system for recording calls, and they would 18 follow scripts to determine -- initially, there were no 19 scripts but we developed scripts later, as the Helpdesk 20 matured and as problems emerged, to try and determine 21 where the postmaster or the operator of the counter 22 terminal was in relation to the process of performing 23 a transaction and what had gone wrong at that point. 24 So the Helpdesk, that became the first line of 25 support. 7 1 There was a second line of support, which dealt 2 with more hardware-type problems like -- so if the comms 3 had gone down or if the barcode reader had failed or the 4 printer hadn't worked, then those things would be passed 5 on to the second line of support who -- they would 6 schedule an engineer mostly to deal with that problem. 7 Then anything that was a little more complicated 8 that couldn't be resolved in that way and with 9 particular timescales, as well -- I can't remember what 10 they were particularly, but there were quite stringent 11 timescales in which we had to resolve these issues -- 12 then the call would be escalated to third line, which 13 would be the SSC, the Systems Support Centre, and they 14 had far more knowledge of the application itself, not 15 from a development perspective but they had access to 16 how the system operated and they knew how the system 17 software integrated with the hardware, and so they would 18 be able to deal with a much more in-depth query and 19 hopefully resolve a fault. 20 If they couldn't resolve it, then the problem 21 rested with development. So there was something 22 fundamentally wrong with the product and it would be 23 escalated to fourth line support. But the HSH, 24 primarily, was levels 1 and 2. So the HSH taking calls 25 direct from the postmasters and then passing on to the 8 1 SMC, the second line of support, and they were mostly 2 engineering calls. 3 Q. Did you hear Kevin Fletcher's evidence from earlier this 4 week at all? 5 A. Kevin Fletcher? 6 Q. Yes. 7 A. No. 8 Q. One issue that was addressed was training and his 9 evidence was that any concerns or concerns about 10 training and the length of training -- so let's say it 11 was a day and a half for managers -- would have been 12 resolved because there was a Helpdesk, so users could 13 use the Helpdesk. Did you see the Helpdesk as 14 fulfilling that kind of a role, filling the gaps in the 15 training? 16 A. At the time, no. Now even less, I think. 17 Postmasters took many years to get where they were 18 in dealing with the processes and procedures of the 19 Post Office; selling stamps is not as trivial as it 20 sounds. But even more so, when you start introducing 21 benefit encashment services and those things were very, 22 very complicated and, even the postmasters struggled 23 before in my understanding before Horizon with some of 24 the rules, and so on. But at least at that stage they 25 were in charge of everything themselves. 9 1 When it went into Horizon, the recording of what 2 they did was assumed to be automatic and sometimes the 3 software may have assumed that they did things as the 4 software expected them to do and, if they didn't, then 5 there would be a problem. The Helpdesk had the dilemma, 6 it didn't understand, first of all, how the Post Office, 7 how the postmasters did their normal operations. They 8 were simply responding to "I've got a problem with my 9 printer" or "I can't balance" or something like that. 10 They would follow a script but they didn't -- that was 11 in no way a substitute for 20 years' experience of doing 12 that type of thing manually and so I don't think the 13 Helpdesk was capable of doing that. 14 Q. In terms of the training then, were you involved in the 15 training in the early stages? 16 A. No, not involved in the training. I was involved in 17 negotiating with Peritas and I think earlier it was 18 called ICL KnowledgePool. 19 Q. Yes. 20 A. I think there were three names -- ICL Training Services, 21 KnowledgePool and then Peritas -- which they were the 22 professional trainers and we simply negotiated contracts 23 with them and they learned their input from Post Office 24 Counters Limited. 25 Q. This is slightly out of order but, just reflecting on 10 1 that training, do you think it was sufficient, given 2 your experience of subsequent issues with the Helpdesk? 3 A. Well, I wouldn't have said it was totally inadequate. 4 That's a very loaded criticism. But how can -- it was 5 sufficient to talk through the process of operating the 6 equipment to perform a specific transaction. Where 7 I think it failed and was not adequate was that you 8 couldn't imagine the sort of things that a postmaster or 9 member of the public had done even to disturb that 10 perfect expectation of the software. 11 So software is written to some rules and the rules 12 are that you do this, this, this and these are the sort 13 of -- you've seen them, the sort of drop-down options on 14 a spreadsheet, for instance. If you do something that's 15 not there, then it's the lack of robustness of the 16 system that causes the problem and I think it was not 17 clever enough to anticipate all of the different ways in 18 which the operators and the environment, you know, 19 communication systems included, could perversely affect 20 their sort of ideal expectation of events. 21 Q. I'm going to move back in time and talk about the early 22 stages, early issues, Acceptance Incidents. We've heard 23 a lot about those in Phase 2 and I won't spend too much 24 time on them, but there appear to be three particular 25 Acceptance Incidents during the contractual stage that 11 1 you were involved in. 2 Can we look at FUJ00119869, please. This is 3 a note from an acceptance workshop on 9 September. Do 4 you remember what acceptance workshops were at all? 5 A. Yes, yes. 6 Q. Can you tell us very briefly what their purpose was? 7 A. Basically, Acceptance Incidents were things that got in 8 the way of Post Office paying -- accepting the system 9 and paying Fujitsu for what it had done. So there were 10 some very strict rules of -- I can't remember precisely 11 what they were but you had to have zero of these and no 12 more than one of those, and so on, and these Acceptance 13 Incidents were those keenly discussed at these meetings. 14 Q. We have your name there down as a representative of 15 Pathway and we have three numbers after your name 408, 16 412 and 298. I will use this document just to refresh 17 your memory as to what those were. 18 If we look at page 3, we have there 298 was 19 "Systems Stability". Do you remember systems stability 20 being an acceptance issue? 21 A. Yes, yes. 22 Q. Very briefly, are you able to remind us what that was? 23 A. Well, things would go wrong without any clear 24 explanation at the time. There might be a blue screen, 25 which I remember that this was a Windows NT system and 12 1 Windows NT was notorious for blue screening; things 2 would go slowly; there would be a scheduling problem 3 within the software; when the system for -- unexpectedly 4 simply didn't work properly. 5 Q. Can we look at page 13 which addresses Acceptance 6 Incident 408. It has there "408 [Horizon System 7 Helpdesk] Performance". Can you remember in brief terms 8 the issue there? 9 A. Yes, yes. I was very, very disappointed with the 10 Horizon System Helpdesk performance not because they 11 weren't trying but because we couldn't get the right 12 staff, the right quality of staff to stay in the 13 Helpdesk. This was part of the work that was contracted 14 out to another division in ICL and it was always the 15 case that the Helpdesk was blamed for something, whether 16 they'd given false information or wrong information -- 17 not false. Sometimes I would say they gave misleading 18 information. 19 There were a number of occasions when I felt that 20 the Helpdesk was not performing as it should and 21 I think, in fact, we raised two red alerts on the 22 Helpdesk. 23 Q. We will come to speak about those red alerts in 24 a moment. Can we look over the page to incident 412. 25 That's described as "Service Performance Ad Hoc 13 1 Reporting". Do you remember that at all, very briefly? 2 A. I don't remember it from the top of my head. I'm just 3 reading it again. 4 Oh, ad hoc reporting, yes. This was the situation 5 where we felt -- my MIS team and the business support 6 unit team felt that Post Office were being a little free 7 with their requests for information and they were 8 demanding things, ad hoc reports, and I think we were 9 probably snowed under, just keeping ahead of -- or 10 keeping abreast, not ahead -- of what we had contracted 11 to do. I think we had underestimated the volume of Post 12 Office asking for analysis of data, and so on. 13 Q. Thank you. I want to focus today really on 408 and can 14 we look at POL00028468, please. This is a plan for the 15 resolution of 418 and it's dated 8 September 1999. 16 That's the top right-hand corner. Do you remember this 17 at all? If we scroll down, it has your name as somebody 18 who reviewed the document. Is this something that you 19 remember? 20 A. Sorry, I will remember the words when I read them again 21 but it's not something that's sort of fixed in my mind, 22 no. 23 Q. Let's look at page 5, please. It's the bottom of page 5 24 and it sets out there the Post Office's position. I'll 25 read those briefly for the purpose of the transcript and 14 1 to refresh your memory. It says: 2 "Based upon the minutes of the Acceptance Board 3 meeting of 18 August 1999, POCL contended that: 4 "'Production of scripts is not complete'. 5 "'It does not take account of activities such as 6 the need to train staff'. 7 "'Some items have already missed dates'. 8 "'Call volume projections and staffing projections 9 contain assumptions that POCL cannot agree based on 10 experience to date'." 11 Then it has some further points just over the 12 page: 13 "POCL's experience to date is that some scripts 14 have resulted in inappropriate advice resulting in 15 further calls to HSH and the [NBSC]. 16 "POCL requires an explanation of how the call 17 volume projections are produced and the logic that 18 supports this process. 19 "POCL requires that the SLA rectification plan is 20 produced and agreed." 21 Do you remember those concerns and do you remember 22 whether you agreed with them, disagreed with them, had 23 a concern about that? 24 A. Frankly, I don't think there's anything to disagree 25 with. These are all truisms. I didn't necessarily 15 1 understand at the time how many of these things. When 2 we put together our call volume estimates, the plan for 3 sizing of the various services, some of it was a shot in 4 the dark and missed. So we had to come together and 5 produce a rectification plan, which is what this is all 6 about. 7 Q. Can we go back to the workshop of 9 September. So 8 that's after this. So that's FUJ00119869. If we look 9 at page 13, I'm going to read to you that first entry. 10 It says: 11 "Pathway will arrange a workshop aimed at giving 12 POCL confidence in their resourcing model and to confirm 13 their analysis that a level 3 expert domain for cash 14 accounting is required. Report back outcome and issues 15 to this group." 16 Do you remember the issue about requiring 17 a level 3 expert domain and what that might mean? 18 A. Not specifically, no, but it seemed sensible. 19 Q. So you've described to us the various levels of 20 Helpdesk. I that saying that there should be extra 21 expertise in relation to cash account issues? 22 A. Yes, in the SSC level 3. 23 Q. Do you remember why that might have been needed at that 24 time? 25 A. Because the first and second level support structure was 16 1 inadequate to be able to resolve those issues and it 2 would inevitably be escalated to level 3 and, if you 3 didn't have more expertise in there then where else? 4 Well, you would have to escalate it to level 4 and they 5 were doing development of the next release. So, no, we 6 had to have level 3 expertise. 7 Q. Can we go to FUJ00119870. This is a bit later on, so 8 13 September, not too far on. Can we look at page 11. 9 We return there to Acceptance Incident 408 and, again, 10 on the second entry there: 11 "Pathway to produce outline proposal on Service 12 Levels for the cash accounting expert domain." 13 Do you remember that ultimately happening, the 14 extra assistance for cash accounting? Did it happen? 15 Is that something you have any recollection about? 16 A. It must have done. I can't specifically remember it 17 from an event flag that -- I just don't remember that 18 but it must have done because, ultimately, this was 19 resolved. 20 Q. I'm going to read to you that final -- 21 A. Excuse me, and ultimately we did have more expertise in 22 level 3 in the SSC. 23 Q. Do you remember when that was? Was that on -- 24 A. No, no, but it would be within this time period 25 certainly. 17 1 Q. If we look at the final entry on that page, it says: 2 "Performance Service Level statistics for August 3 have been reproduced by Pathway to exclude the cash 4 account calls. POCL to assure that the statistics are 5 being appropriately reported. Pathway and POCL ... to 6 meet to review the new service level report." 7 Is that something you remember at all? 8 A. Yes, I do. I mean, I remember Dave McLaughlin and Ruth 9 Holleran saying "Well, we've got to make sure that you 10 have not bundled a lot of other stuff in with cash 11 account". So what we were trying to do here was show 12 that the performance of the Helpdesk had improved and 13 the performance of the system had improved without the 14 effect of the cash account calls. So this was -- if you 15 consider the cash account was very special and difficult 16 topic, how were we doing on the rest of them, and that's 17 the purpose of that activity. 18 Q. Thank you. You have said that the cash account is 19 a difficult topic. Can you expand on that for us 20 a little bit? 21 A. I wish I could. Cash account, stock units, the transfer 22 of stock from unit to unit, I wonder sometimes if it's 23 just too complicated. Clearly, I mean, I think it's 24 probably been resolved by now. I don't know. I've not 25 seen what Horizon -- what's the new one? 18 1 Q. Online. 2 A. I don't know if that has resolved the problem but it was 3 certainly very, very complex for the Legacy system. 4 Q. Do you remember at that time being told about particular 5 problems with the cash accounts? 6 A. I remember being -- well, I remember there was a problem 7 with -- if you had voided a transaction but hadn't meant 8 to or had not allowed it to print, then there would 9 be -- it would be left in a funny state and, for 10 instance, I think you could pay a benefit twice because 11 the system didn't think it had been paid but, in actual 12 fact, you had handed over the money and that, for 13 instance, would be a difficult thing. 14 I think there was another issue in small numbers 15 of offices -- sorry, small numbers of counters in 16 an office where they might have -- a different counter 17 clerk would have his own stock unit but stock had to be 18 transferred from the previous counter clerk's -- a bit 19 like shift work and you have got to transfer. So I've 20 got 100 stamps left and I've got to transfer those 100 21 stamps to a different stock unit. 22 If that hadn't operated exactly as the software 23 anticipated, then there would be a problem. 24 Q. Were those kinds of issues well known within ICL at the 25 time or not? 19 1 A. Well known within -- not within ICL. Within Pathway, 2 yes. 3 Q. Within Pathway, sorry, yes. When you say not within 4 ICL -- 5 A. Well, no, ICL Pathway was separate from ICL. 6 Q. If I could just take you back to the first document we 7 looked at, so it's POL00028211. On that first page 8 that's the overall Pathway board and you have Mr Bennett 9 there -- 10 A. Yes. 11 Q. -- and you have Mr Christou from ICL -- 12 A. Yes. 13 Q. -- and Mr Todd from ICL, they all reporting to the 14 Chairman, Sir Michael Butler. Were those kinds of 15 issues, as far as you were aware at the time, the kinds 16 of things that would be discussed with ICL? 17 A. Not in that granularity. Certainly, the board would be 18 very interested in how we were doing, how we were 19 performing in meeting the service level agreements. 20 I mean, once it moved from a Private Finance Initiative 21 where Pathway had all of the liability to an ordinary 22 contract, then there were very, very specific targets to 23 be met and failure to meet, say, those targets meant 24 financial penalties on ICL Pathway and, therefore, on 25 the board. They were certainly made aware of how well 20 1 we were doing or how badly we were doing because, 2 indeed, we did suffer penalties. 3 But they wouldn't have known in such fine detail 4 the reasons for those things. 5 Q. Can we look at POL00028509. This is on the same theme 6 as the documents before. This is a 14 January meeting 7 in 2000 -- sorry, this is forwarding it but, if we turn 8 over the page, it refers to it as a "Special Meeting" at 9 Gavrelle House. Do you remember that meeting at all? 10 A. Sorry, no. 11 Q. This seems to have been a meeting to decide on the 12 recommencement of rollout and, if we look over the page, 13 there's a section that I can read to you at the top of 14 the page. It says: 15 "Tony Oppenheim advised that ICL Pathway intended 16 to move forward with POCL on the contractual agreement 17 immediately following the meeting. The meeting between 18 Andy Radka and Steve Muchow earlier in the day on the 19 outstanding issues surrounding [Acceptance Incident] 20 408/3, and the level of agreement that had been reached 21 would facilitate this contractual discussion. It was 22 and intended that the summary of actions that had been 23 produced as a result be incorporated as a working 24 document, following review by the lawyers of both 25 parties." 21 1 So it seems as though there was agreement on that 2 date to essentially go ahead with Horizon, despite 3 issues with Acceptance Incident 408 still continuing. 4 Do you remember that at all? 5 A. I don't remember the degree to which the outstanding 6 issues with 408 impacted Post Office's perception of the 7 viability of continuing the rollout but, clearly, we did 8 continue the rollout and so I must assume that we'd come 9 to an agreement that it was okay. I can't remember the 10 details. 11 Q. I'll just read the final paragraph on that page. It 12 says: 13 "Agreed that if actions in place to address the 14 outstanding elements of agreement worked, and no further 15 issues arose prior to signing the third supplemental 16 agreement, there was no requirement for a further 17 meeting." 18 Do you remember agreement to work on the issues 19 relating to Acceptance Incident 408 going forward? It 20 hadn't come to an end in January at the time of rollout 21 or just before rollout? 22 A. They never came to an end is the honest answer to that. 23 I mean, these things -- it's a matter of degree and 24 risk. The customer needs to decide what he's prepared 25 to accept in terms of risk, quality of service, 22 1 performance, and so on, and if we had been able to 2 persuade them it was acceptable then they would go 3 ahead. But they would need to make that decision for 4 themselves. It was not -- we couldn't insist. So we 5 worked constantly to try and improve. 6 These things -- there will never be zero. You 7 notice in some of the requests for performance there's 8 a target level of zero. Well, I'm sorry, but we never, 9 ever achieved zero, not -- except by good fortune in one 10 particular month. It's an exponential curve approaching 11 zero, the more mature that the product becomes and the 12 more experience that the support teams and the users 13 have in the characteristics of the product itself. 14 So I think what this is saying is that we did come 15 to an agreement that it was down to a sufficiently 16 manageable level that didn't pose a risk to going 17 forward with the rollout. 18 Q. Can we look at POL00028512. This is very shortly after 19 and it's before the rollout resumes again in January. 20 This is sent to you by Paul Westfield. Do you remember 21 who Paul Westfield was at all? 22 A. Oh yes. I'm his son's godfather -- well, I wish I were. 23 Q. What was his role? 24 A. Paul was in charge of a number of things, actually, to 25 do with managing the service delivered. 23 1 Q. Can we turn over the page, please. 2 A. I can't remember his job title, to be honest. 3 Q. So this is, again, "Acceptance Incident 408: Cash 4 Account Call Analysis Review -- Week 1 & Improvement 5 Plan". 6 A. Yes. 7 Q. Perhaps we could turn to page 6. I'm just going to read 8 to you that introduction. It says: 9 "In accordance with the monitoring 10 requirements ..." 11 So it seems as though there were monitoring 12 requirements going forward for Acceptance Incident 408? 13 A. Yes. 14 Q. "... the [Horizon System Helpdesk] sites at both 15 Stevenage and Manchester are recording all Cash Account 16 calls for a six-week period from [3 December 1999]. The 17 taped calls are then being reviewed by POCL who will 18 make an assessment as to the [Helpdesk's] ability to: 19 "Conform to the narrative contained within the 20 Cash Account scripts. 21 "Give out correct advice avoiding a negative 22 impact on the POCL business." 23 So this seems to be along the lines of what you 24 have just discussed, which is that POCL would be 25 monitoring the progress going forward? 24 1 A. Yes, and those calls were recorded and they were 2 reviewed. They were. 3 Q. If we look down at the bottom of that page, it gives the 4 initial results. It says: 5 "POCL reviewed 45 calls out of 177 recorded for 6 Cash Account activity on 08 & 09 [December] 99. Out of 7 the calls reviewed, 13 were deemed to have failed in 8 that by incorrect advice being given by the HSH this 9 could have a negative impact on their business, or the 10 HSH deviated from the Cash Account script." 11 Is that something you remember? 12 A. Not specifically but, yes, that's the sort of thing. 13 Q. If we turn over the page, there is a table there. It 14 seems as though there's a difference of opinion between 15 POCL and ICL as to how many failed or not. If you look 16 the second line, "Number of Calls Failed": POCL after 17 Initial Review, 13; POCL after Joint Review they came 18 down to 8; ICL Pathway view after Joint Review was zero. 19 So it seems as though there's quite a significant 20 difference of opinion as to what amounted to failure. 21 A. But after joint review there's a considerable coming 22 together of minds. 23 Q. Sorry, can you just expand on that? 24 A. Well, POCL view after Joint Review, five number of calls 25 passed and only eight failed not 13. So ... they'd 25 1 moved their position. They were persuaded that it was 2 not necessarily just the Helpdesk at fault. I think the 3 scripts were largely to be examined to see whether or 4 not they went far enough. I think there was -- the 5 scripts sort of ran out of steam. I think there's some 6 talk of that later on in this document. 7 Q. Shall we look at page 10 which is the improvement plan 8 and I'll just read to you halfway down that first 9 paragraph. It says: 10 "The components of this improvement plan have to 11 be developed, tested and implemented within the 12 [Helpdesk] prior to the expected commencement of rollout 13 on [24 January 2000] 14 "From the 13 calls analysed in this joint review, 15 and from experience gained within the ICL Pathway 16 Customer Service Management Information Reporting, 17 specific areas can be identified as causing confusion 18 either in the outlet or at the HSH, these are believed 19 to be: 20 "1. Out-of-hours stock units (eg Lottery) and 21 associated prize allocations. 22 "2. Discrepancies and dealing with the entire 23 complex subject of reversals and suspense accounts." 24 So this is something that you had briefly 25 addressed before -- 26 1 A. Yes. 2 Q. -- about a particular issue with discrepancies and cash 3 accounts. Can you perhaps expand on the significance of 4 that? 5 A. I think what this tells me now is that we should have 6 recruited postmasters who knew what they were talking 7 about to do this role to help postmasters and, in fact, 8 later on, with the introduction of the -- what was it 9 called -- the Network Business Support Centre, which was 10 another helpdesk manned by Post Office Counters Limited, 11 these issues were dealt with there. I think that was 12 a sort of admission that lay persons simply couldn't 13 handle that type of call with good effect to the 14 satisfaction of Post Office. 15 Q. Can you assist us with that, actually about, the role of 16 the NSBC (sic) -- 17 A. NBSC. 18 Q. -- NBSC -- and how that fit in, both at this time and, 19 as you said, later on? 20 A. I'm not sure whether the NBSC -- yes, it is: 21 "Where a business rule needs to be invoked by the 22 NBSC." 23 So the NBSC was equivalent to the HSH for 24 postmasters for Post Office-related things and it 25 ultimately -- I think it took on, if not all, a lot of 27 1 the work to do with dealing with cash account balancing, 2 and so on, problems that we had not been very good at. 3 But the NBSC was -- it provided support to the network 4 and the postmasters were their staff, if you like. 5 Q. Where did you see software issues that caused issues 6 with balancing to fit into that overall picture of help? 7 A. Well, they wouldn't be resolved there. They wouldn't 8 even be identified there. They would be identified in 9 third line support software issues. We had a number of 10 systems. I think you've heard of the KEL. 11 Q. The Known Error Log? 12 A. The Known Error Log. It's more a font of all -- it's 13 somewhere you could dump useful information 14 a sophisticated Frequently Asked Questions-type affair. 15 They could look in there and that would -- they 16 should be able to, or they should have been able easily 17 to have found that this was a common issue, that 18 somebody else had had this problem. 19 When somebody has a problem for the first time 20 you're on your own. I mean, everybody's -- we don't 21 know. When a problem arises for the first time you're 22 in discovery mode. 23 When it arises for the second, third, fourth, 24 fifth, 25th time, then you know you've got an issue 25 which is potentially an operational issue, 28 1 an infrastructure issue, a software issue. All of these 2 things can come together to make it fail. 3 Interestingly, there was a time -- I remember when 4 we had -- oh, I think it was in NR -- New Release 2. 5 There were a number of sites had no issues at all and 6 some sites had terrible problems balancing. There is 7 a document in my original bundle which demonstrates 8 this. 9 The assumption was that there was a fault on the 10 network that was dealing with that place but, in fact, 11 I don't think it was a network fault. I think it was 12 something that had simply maybe have been missed or 13 miscommunicated in the training and this group of 14 postmasters who were doing it differently to this other 15 group of postmasters. So my question was "Well, why 16 have these guys got problems and these guys haven't?" 17 There's something markedly different between the two 18 groups, and that's where you need the SSC to delve in to 19 find out precisely what was going on and to see what the 20 root cause was. 21 If they could fix it -- they couldn't fix it per 22 se with a software fix but they could pass that on to 23 development and they could look at it and see whether or 24 not it was reproducible on their test rigs and, if it 25 were, then they could incorporate that into the next 29 1 change for the next release or a maintenance release. 2 Again, we'd have to discuss that with Post Office but 3 that's how the system worked. 4 So the Helpdesk itself would only basically know 5 either what was in the script or what had been reported 6 before that had been fixed with a known workaround or 7 a reinforcement of procedure. 8 Q. So you spoke about the Known Error Log. Who had access 9 to that? 10 A. I believe just the SMC and the Helpdesk and I think 11 fourth line would have done but they were more 12 interested in PinICL. 13 Q. Would the Post Office have had access to it? 14 A. I don't believe so. They might have done. They did 15 when they had staff in Feltham working alongside the 16 test teams. So, yes, they would have had access then. 17 Q. Who were those teams? 18 A. Sorry, which? 19 Q. From the Post Office? 20 A. I don't know. Probably -- 21 Q. What was their job, though, in Feltham? 22 A. They would be looking at model office rehearsal and -- 23 yes, model office rehearsal, I think. MOR1, MOR2 from 24 recollection. 25 Q. Your understanding is that they would have had access to 30 1 something -- to the Known Error Log? 2 A. Perhaps that's too strong. Maybe I should say they were 3 not denied access but it was there and -- 4 Q. So they would have access to ICL internal network or 5 internal systems? 6 A. I don't think they will have had access to internal 7 systems, no, because these were shared systems sometimes 8 and had information on them which wouldn't have been 9 right to share with the outside. 10 Q. So something like the Known Error Log may have been 11 something they could have requested, for example; is 12 that your evidence? 13 A. I think they could have done, yes. 14 Q. But it's not something that they would have made 15 available to them as of right? 16 A. I cannot recall mandating that Post Office should have 17 their own access to the Known Error Log but I don't 18 believe they were ever denied access to that and I'm 19 pretty sure that PinICL and Known Error Log was used in 20 communication with Post Office when we were discussing 21 problems and, in fact, in many of the boards, I think 22 even in the CAPS board, there were PinICLs discussed 23 there. 24 But the PinICLs were likely to have been, if you 25 like -- I don't like using the word "sanitised" because 31 1 it suggests we're hiding something but there would have 2 been extracts exported from PinICL to give to the 3 management teams who were deciding when to go forward or 4 whether not to go forward. 5 Q. Thank you. I'm going to take you to another document. 6 FUJ00118186. 7 This is the third supplemental agreement that was 8 between Post Office and ICL Pathway on 19 January 2000. 9 Is this something you had any involvement with? 10 A. I think I was involved in meetings but I'm not sure 11 I can remember -- 12 Q. If we go to -- 13 A. They were very dry meetings! 14 Q. Page 7 of that agreement is schedule 1 and it concerns 15 Helpdesk improvements. 16 A. Yes. 17 Q. Do you remember Helpdesk improvements being 18 a significant part of that agreement? 19 A. Oh, yes, yes. 20 Q. If we look at, for example, "Call Scripts", it says 21 there: 22 "The Contractor and POCL agree that separate call 23 scripts shall be introduced to be followed by Helpdesk 24 staff in relation to: 25 "out-of-hours stock units ... and 32 1 "discrepancies and dealing with reversals and 2 suspense accounts." 3 Those were the two concerns that we spoke about 4 just before -- 5 A. That's what we were discussing earlier. Yes, it is, 6 yes. 7 Q. If we look at -- 8 A. I mean, we drafted them and, as this says, POCL reviewed 9 them and said they were okay or not. 10 Q. Can we look at page 9, please. It goes through other 11 agreed improvements and one of them is the "Horizon 12 Guide to Balancing", and it says: 13 "The Contractor shall review all cash account 14 scripts in use at the date of this Agreement and shall 15 ensure that they are consistent with the guide produced 16 by POCL (and provided to the Contractor prior to the 17 date of this Agreement) called 'Balancing with 18 Horizon'." 19 A. Yes. 20 Q. Is that something you remember? 21 A. Yes. I don't remember the content of it but I remember 22 that specifically because what we wanted to do was "To 23 get a definitive statement, this is what you should be 24 doing", and that's what we hoped Post Office provided. 25 Q. This was all shortly before the national rollout resumed 33 1 on 24 January. What do you recall of the Post Office's 2 attitude towards those kinds of issues, the Helpdesk 3 issues and the issues that you were involved in? 4 A. Well, Post Office's attitude was always one of getting 5 the best for Post Office from the contract. I mean, 6 these guys were quite good at driving a hard bargain. 7 It was my job to staff up the people and manage the 8 information, management information, which enabled us to 9 see how well we were doing and to persuade Post Office 10 because -- of how well we were doing because in that 11 distillation of information resided the reward. I mean, 12 we were paid a sum of money but then we had to give back 13 for all of the failures that we had and so it was in 14 Post Office's interest to make sure that they were very 15 well documented on what we had to do and that they were 16 assured that when we said we had done something, that it 17 had been done because, if we hadn't done it, then we 18 would owe them some money. 19 I think it's a typical contractual relationship. 20 Q. We've seen there, in terms of required Helpdesk 21 improvements, focus on discrepancies and focus on 22 balancing. 23 A. Yes. 24 Q. Were those issues quite prominent issues in your 25 discussions? 34 1 A. I think particularly on balancing, yes. Balancing was 2 a big issue. Cash account discrepancies, a big issue. 3 I cannot imagine -- I mean, the whole business revolved 4 around selling products for themselves and for other of 5 their clients and they had to level up, they had to 6 settle up at the end of the month, or whenever, and so 7 it was important that the information was correct. 8 Q. What was the atmosphere like? Was there anger, upset? 9 A. No, no. I mean, I think irritation sometimes but 10 I think we tried to do business in a professional way. 11 We didn't fall out about it. But we didn't get our own 12 way and we had to fight for every improvement that we 13 thought we'd made. 14 There were things, for instance, that the Post 15 Office did that made life difficult for us. I mean, 16 consider reference data. If you issue reference data to 17 the post offices and say that 10 penny stamp has changed 18 to become a 10 penny stamp, and that's what happened. 19 So there was huge volumes of reference data that we had 20 to process unnecessarily and that degraded our 21 performance capability and we possibly hadn't allowed 22 for that level of work in our assumptions of the volumes 23 early on when we struck the contract. 24 So, yes, there were -- there was some give and 25 take to be had pointing these anomalies out and trying 35 1 to do a quid pro quo, I guess. 2 Q. I'm going to go on to talk about technical issues, 3 software issues with Horizon. Was the link ever drawn 4 between these issues that postmasters were having and 5 the Post Office was recognising on the Helpdesk, insofar 6 as balancing is concerned and technical issues -- 7 A. Yes, they -- that's why they took to insisting on the 8 new scripts, validating those scripts and recording the 9 conversations. So, yes. 10 Q. Were those aimed at improving the way in which 11 a postmaster would go about using the system or were 12 they aimed at identifying actual technical problems with 13 the system itself? 14 A. I don't think the postmaster can do any more than follow 15 his instructions and, when things go wrong, report 16 a problem and that problem to be escalated through the 17 support chain to -- eventually to become an incident 18 which is recorded on PinICL and then resolved by 19 a software change. Hopefully, there might have been 20 a workaround to mitigate his situation at the time and 21 to keep things moving but I don't think that the 22 postmaster could have done any more than that. 23 Q. As the person who was responsible for the Helpdesk, you 24 were seeing these workarounds for example, being put in 25 place and you, at the same time, were being blamed for 36 1 failing to meet certain objectives. 2 Was there ever a thought in your head that 3 actually the problem is the software, rather than the 4 Helpdesk? 5 A. When the problem was the Helpdesk, I sorted out the 6 Helpdesk -- well, except I didn't. I sorted out the 7 contractor for the Helpdesk. When the problem was the 8 software, then we sorted out the software. 9 There was no problem within Pathway between 10 customer service and development identifying problems. 11 All developers want their products to be as good as they 12 can be. It would be lovely to have -- impossible but it 13 would be nice to think that one day there would be no 14 need for a Helpdesk. You know, that things don't go 15 wrong but they did and they will and they continue to go 16 wrong. 17 Q. So was it always envisaged that there would be these 18 software issues and that was the purpose of the 19 Helpdesk? 20 A. No, the purpose of the Helpdesk was to help the 21 postmaster operate the system. It also -- I mean, to 22 capture complaints to -- whatever call came in -- it 23 could have been a member of the public in the early days 24 with the Benefits Encashment Service. A member of the 25 public could call the Helpdesk and say that they had not 37 1 been able to pick up their Benefit Payment Card or 2 whatever. 3 So it was the first point of contact to gather 4 together all the things that were wrong, as perceived by 5 the operators of the system, the postmasters, the 6 members of the public, the users, and we occasionally 7 got calls from Post Office as well. Anybody could call 8 the Helpdesk. It was a published number. 9 The filtration of those things and the 10 distillation into specific problems that were capable of 11 being fixed by changing the software was the job of the 12 System Support Centre and development and the test 13 teams. We had test rigs in the SSC that could reproduce 14 the fault. If we could reproduce the fault we were 15 happy because then we could show concrete evidence to 16 development, "Here, chum, you've got a problem here, fix 17 it", and that was always the best way. 18 The intractable problems the ones where we 19 couldn't reproduce it. I'll give you an example. On 20 communications faults, there were several occasions when 21 comms would go down and miraculously return; nobody had 22 done anything. So it was -- it's an amalgam of skills 23 and effort and expertise to try and resolve issues and 24 get them fixed as soon as we can. You can't simply fix 25 it in the Post Office at the time because it's an estate 38 1 and you have to do a rollout. 2 We did occasionally put a fix to a specific post 3 office but then that would have been overwritten by the 4 next -- that would be there to say "Did this actually 5 cure the fault as seen by the postmaster?" But that 6 then would have to be incorporated into a change, a new 7 release -- a maintenance release or a new release to 8 affect the whole estate. 9 Q. Can I ask you how that happened. So to an individual 10 terminal, for example -- 11 A. To an individual? 12 Q. Terminal. 13 A. Terminal. 14 Q. How would you go about making that fix? 15 A. Well, there wouldn't be -- this is a bit technical for 16 me but there wouldn't be an individual terminal except 17 in single-counter offices where we had then an extra 18 disk which effectively -- because we always had a backup 19 of the message store and then there was a copy on the 20 correspondence server in our data centre. 21 So we would have to make a connection and, on 22 occasion, particularly, say, for instance, when we had 23 a communications fault, we would open the connection 24 from the data centre and keep it open, so that we could 25 put down a fix to the PC, which was the counter and if 39 1 it were a multi-counter office that would then be 2 replicated -- I can't remember the term but it would be 3 propagated to all of the counters in that post office 4 and -- 5 Q. Who would do that? Who was responsible for doing that 6 task? 7 A. The only people that could do that would be the SSC and 8 development working together. 9 Q. Could they do that to, for example, a cash account? 10 A. In what way? 11 Q. Could they implement a fix that might impact on the cash 12 account to a single post office? 13 A. Yes and no. They could make a change to -- a balancing 14 change but it would be a new transaction. It wouldn't 15 be -- I don't believe they could alter a transaction. 16 They could put in a new transaction. So, for instance, 17 there was -- how can I put it? 18 I'm just running out of my comfort zone here but 19 I think, if there was an imbalance, they could insert 20 a balancing sum to correct that so that the postmaster 21 could rollover to the next cash account period and carry 22 on work. I mean, this was a requirement because, 23 otherwise, he would be stuck. He couldn't do any 24 business. And we would do that with the knowledge of 25 Post Office, with the NBSC, that that's what we were 40 1 doing. 2 In fact, I think they had to agree that process 3 because Chesterfield had -- I can't remember the name of 4 it. Is it TPS? Transaction ... there's a -- 5 Q. TIP? 6 A. TIP, maybe it's TIP. There was a Post Office Counters 7 Limited system that would read in all the transactions 8 and it would get one which would -- we had to tell them 9 why we'd done that. So I think that's how it worked, 10 yes. Yes. 11 Q. I'm going to ask you about technical issues now, insofar 12 as you're able to. You have addressed incomplete 13 transactions in your witness statement and we'll briefly 14 look at those. Can we look at POL00028100, please. 15 Sir, before I move on, is there anything that you 16 wanted to ask in relation to that access point? 17 SIR WYN WILLIAMS: No, thank you. But since we've got this 18 very short break in your line of questioning, in about 19 ten minutes could you engineer a short break for me, 20 please? 21 MR BLAKE: We will take a ten-minute break today if that is 22 sufficient. 23 SIR WYN WILLIAMS: Yes, that's fine. 24 MR BLAKE: Can we look at page 146 of this document, please. 25 So we're moving back in time now to 1998, I'm afraid, 41 1 and this is a time when the Benefits Agency was still 2 very much involved. You'll see this is a letter to 3 Mr Vince Gaskell of the CAPS programme and it's a letter 4 from yourself dated 15 September. 5 A. Yes, I remember this. 6 Q. I'm just going to read that final paragraph. It says: 7 "You may note that as overall transaction rates 8 increased, the problem diminished. In August, the 9 success rate was 99.98% -- with less than 3 transactions 10 per 10,000 being incomplete." 11 Do you remember approximately how many 12 transactions might take place in a day or a week or ... 13 A. No, I'm sorry, not offhand, no. 14 Q. "Our target is to continue to reduce the number of 15 incomplete transactions towards zero and we are 16 confident that where the cause is a systematic error or 17 where a systematic preventative measure can be devised 18 then this will be achieved." 19 You said there "towards zero" and that's 20 an important point that you raised in your evidence 21 earlier, that you will not get to zero; your aim is to 22 go towards zero. Have I understood your evidence 23 correctly? 24 A. Absolutely, yes. 25 Q. "There will always remain a residual 'human element' for 42 1 which there is no ready answer except that with 2 increasing experience of the behaviour of our end-user 3 community we will be able to reinforce the application 4 of correct operational procedures through focused 5 feedback and transaction re-engineering." 6 A. Yes. 7 Q. Can you tell us a little bit about that and what you 8 meant by that? 9 A. Yes. This was very dear to my heart. 10 What I was striving for was, if you like, hostile 11 testing. Human beings don't always do what they're told 12 to do and programmers always -- well, they are supposed 13 to -- always do what their specification says they must 14 do. So when a program is written to say "Take the 15 numbers out of these three boxes, add them together and 16 give me the sum", it expects them to fill in three 17 boxes. Now, imagine one of those things was "Divide by 18 this number" and they'd not filled in that but divided 19 -- and left it as zero, we would have ended up with 20 divide by zero. 21 It's something that we didn't expect because it 22 wasn't written in, it's not a specification for what to 23 do, it's a lack of a specification of what not to do. 24 So it's -- all we can do is anticipate what sort 25 of anomalies might be introduced by the human operation 43 1 of these systems (and we're all human, we all make 2 mistakes, we all type things in in the wrong boxes now 3 and again) and I wanted the system to not fall over in 4 a flap when that happens. And if it's a ridiculous 5 answer, for instance, if it divides by zero and creates 6 an infinity number, then, you know, I don't want the 7 balance to say infinity because that's -- I'm not saying 8 that's what happened. This was something that we tried 9 to get across particularly in the contractual 10 discussions with Post Office about setting targets that 11 were literally never going to be achieved and what's the 12 point of doing that? 13 So, as I say, there's a -- it's like 14 an exponential curve. It approaches zero. You may have 15 periods of months and months and months with no errors 16 at all and think, yes, we've cracked it, but then 17 a spate crops up. So that's what it was about. 18 Q. You refer there to the human element. 19 A. Yes. 20 Q. Is it just the human element -- 21 A. No -- 22 Q. -- that might not make it zero or were there -- 23 A. No. As I said, the human element is that it may have 24 been coded incorrectly. 25 Q. Yes. 44 1 A. Where human beings are involved, there's always going to 2 be errors. 3 MR BLAKE: Thank you. Sir, might that be the appropriate 4 moment to take the ten-minute break? So if we come back 5 at 20 past? 6 SIR WYN WILLIAMS: Yes, that will be fine. 7 MR BLAKE: Thank you very much. 8 (2.12 pm) 9 (A short break) 10 (2.22 pm) 11 MR BLAKE: Thank you, sir. We are back. 12 SIR WYN WILLIAMS: Very good. Thank you. 13 MR BLAKE: Can I bring up on to screen POL00090428, please. 14 This is a very long second supplemental agreement. I'm 15 only going to take you to one page. But is that 16 a document that you were familiar with at the time, the 17 second supplemental agreement? 18 A. I think there was a third as well. 19 Q. Yes. Was it something that you played a part in? 20 A. I might have done if there were changes to requirements. 21 Q. Can we look at page 21. This addresses the TIP 22 interface and I'm just going to read to you that first 23 paragraph. It says: 24 "during the period from 3rd October 1999 until 25 14th November 1999, the percentage of Cash Accounts 45 1 received by POCL across the TIP Interface containing 2 Cash Account Discrepancies shall not exceed 0.6 per cent 3 of all such Cash Accounts." 4 Is that the kind of thing that you were talking 5 about before when you say you can never get to zero so 6 you need to be somewhere above zero? 7 A. Well, it's one example, yes, but even before Horizon, 8 I remember there was a huge department in 9 Chesterfield -- I think there were about 400 staff 10 there -- who were trying to resolve issues with the 11 old-fashioned paper account -- cash account. So, yes, 12 I mean, Post Office had to reduce the cost of that 13 activity and hopefully Horizon would have helped them by 14 eliminating a lot of those faults but, clearly, they 15 anticipated them still being there and 0.6 per cent, 16 I think, is still quite a large number of faults to get 17 through. 18 Q. Was that an acceptance that there would be discrepancies 19 in the cash account going forward, irrespective of how 20 hard either side tried? 21 A. Well, I can't speak -- I'm pretty sure that it was, yes. 22 I can't speak for what they actually felt. I mean, they 23 had aspirations of it being zero. They were running 24 a business and if they could do without some costs then 25 all to the good. 46 1 If it minimised their expenses on dealing with 2 these discrepancies, then, yes. But 0.6 per cent is 3 still a substantial number I would think. 4 Q. I'm going to move to issues post rollout. Can we look 5 at POL00029158, please. This is "Service Review -- 6 Performance Statistics" for January 2000. It's dated 7 7 February 2000 in the top right-hand corner. 8 A. Yes. 9 Q. You were on the distribution list and you are named as 10 the approval authority. 11 A. Yes. 12 Q. Can you tell us what were service review performance 13 statistics or what was a service review? 14 A. Well, if it moved you measured it and I think you see in 15 the next few pages these and ... 16 Q. The names there, are they all ICL or Pathway names? 17 A. They are all my team. 18 Q. They are all your team? 19 A. Apart from Tony, who's contracts director, finance 20 director. 21 Q. Would this kind of a document have been shared with, for 22 example, the Post Office? 23 A. Not in this form, I don't believe. Oh, it might have 24 done with service management review forum, yes, maybe. 25 Q. Can you expand upon that? Why is the service management 47 1 review forum there? What did that mean? 2 A. Well, when we -- we would share our performance with 3 them and they would have to agree and so, yes, it would 4 be shared. 5 I'm not sure whether this document was the one 6 that was shared or whether there was something a little 7 more elaborate. 8 Q. Who formed part of the service management review forum? 9 I don't need names necessarily. 10 A. Well, I think -- well, Andy Radka's name and Ruth 11 Holleran's name come to mind with me and my team, 12 particularly Richard Brunskill, who was instrumental in 13 doing many of these analyses, and Peter Robinson who 14 designed a lot them as well. 15 Q. Can we turn to page 7, please. 16 A. I think -- 17 Q. Page 7 is the management summary. Sorry, did you want 18 to say something else? 19 A. No. 20 Q. This is the management summary. If we look that top, we 21 have there the date of 31 January 2000, 2,000 live 22 outlets and 4,485 operational counters. I'm just going 23 to read to you a few passages from there. So it starts: 24 "As National Rollout recommenced in January, there 25 were 2,000 Outlets in Live operation by the end of the 48 1 month. However, despite the increased number of 2 Outlets, there was a reduction in the total number of 3 calls logged with the HSH (7,017 calls in Jan 2000 as 4 compared with 7,556 calls in Dec 1999). This in turn 5 caused the ratio of calls per Outlet to drop to 3.5 in 6 January, compared with 4.1 in December 1999." 7 Then it goes on to talk about certain issues and 8 I'm going to start with the BT bills issue. It says 9 there: 10 "On 27th January a large number of incidents were 11 raised because BT Bills could not be scanned. This was 12 the result of a Reference Data Process fail and 13 a subsequent overrun during the previous night. 14 "This particular problem was resolved by advising 15 counters of a workaround and transmitting the missing 16 Reference data later that day." 17 So pausing there, you have mentioned issues with 18 reference data. Can you briefly tell us what kinds of 19 issues you had with reference data and whether this is 20 typical or not. 21 A. Well, it's one of many different problems. Reference 22 data is the heart of the configuration of Horizon's 23 system. It basically says what can be sold, where and 24 when and what the parameters of that sale might be, for 25 instance the price of stamps, and so on. For instance, 49 1 not every post office could sell -- not every post 2 office could do a passport, for instance, and so there 3 would be reference data that pertained to that 4 particular post office and sometimes there are quite 5 a number of errors in the reference data. 6 Q. Who provided the reference data? 7 A. The reference data came directly from Post Office 8 Counters Limited. 9 Q. There's a reference there to workaround for the time 10 being until it was resolved. Were workarounds quite 11 common scenarios? 12 A. It's a word -- it's a term that's used quite loosely. 13 It means "How do you get over this problem for the 14 minute", and I don't know of any specific examples. So 15 I'm only guessing, really, but if there were one type of 16 transaction and it was similar to another type of 17 transaction and you had the reference data for one and 18 not the other, you could say to Chesterfield "How about 19 calling it this transaction so they can perform the role 20 but, in fact, it's one of these". 21 I don't know if that's a good example. But 22 workarounds, generally speaking, were not what we were 23 looking for. We were looking for corrections to the 24 reference data. But that meant it had to go through 25 a lot of testing and it could have been a -- it could 50 1 have been a delay that Post Office would not have liked, 2 because of the inability to transact that type of 3 product and that meant a loss of business to the Post 4 Office, to the postmaster and, potentially, to the 5 client as well. 6 Q. Thank you. Then it goes on to refer to an issue with 7 blue screens and you have talked about blue screens 8 already. 9 A. Yes. 10 Q. Then we have "Girobank transaction report". Could we 11 highlight that, please, or blow it up a little so it's 12 a bit bigger. It says: 13 "A report fix was delivered to 1,100 Counters 14 which caused the following scenario to occur in a number 15 of Outlets who were attempting to reverse a transaction. 16 When a transaction was reversed, on a lower numbered 17 Counter node, there was no evidence on the Girobank 18 summary that this reversal had taken place, although the 19 correct information did go to POCL TIP. Some Outlets 20 realised this to be the case and altered the Girobank 21 summary to reflect the correct transactions. Some 22 Outlets however, completed the reversal again, which 23 resulted in a discrepancy for the value of this 24 reversal. MSU have advised POCL of all Outlets where we 25 know a problem has occurred (after calls were received 51 1 by the HSH) and a fix was delivered to the affected 2 Counters on 31st the problem with Giro reports on 3 26th January." 4 Is that something you remember or are you able to 5 assist us with that? 6 A. No. I can't remember the specifics but I do know that 7 it was important that TIP had the transactions in the 8 right sequence and the right counter. So it may have 9 been there case, for instance, that they tried to do the 10 reversal on counter number 3 when it was performed 11 originally on counter number 4 but counter number 4 had 12 failed. Maybe it had a disk error or something, so you 13 have lost a counter or you've lost the communications. 14 So they tried to do the reversal somewhere else and 15 I don't know whether the fault was in TPS or reference 16 data or TIP but that particular impact happened and we 17 discovered it. 18 Q. So we have here, on the page before -- we don't need to 19 turn back to the page -- but it says "operational 20 counters by that stage 4,485", and it said that a fix 21 has been delivered to 1,100 counters. Then we look on 22 this page and it says "MSU have advised POCL of all 23 Outlets where we know a problem has occurred". 24 Now, are you able to assist: would the fixes occur 25 just to those that you knew occurred, so only a quarter, 52 1 let's say, of these counters have been fixed, is that 2 because a quarter would have complained to the MSU or -- 3 A. I really don't know. 4 Q. I mean, let's say that a subpostmaster hadn't called the 5 Helpdesk because they hadn't realised that there was 6 a problem. Typically, would they receive the fix or, 7 typically, would the fix go to those who had raised the 8 issue with the Helpdesk? 9 A. They would eventually receive the fix. I think what 10 would happen is that the Helpdesk would recognise when 11 this had occurred but, if the postmaster hadn't reported 12 the problem, then the Helpdesk would have no record of 13 that and they would not receive the fix until the next 14 maintenance release was distributed to the estate in 15 general. But where this had happened, then what we're 16 trying to do is correct a discrepancy and -- so that 17 that fix would have been delivered to those post 18 offices. But there may well have been other post 19 offices where they failed to -- sorry, not failed. It's 20 not -- where they hadn't reported it and so they would 21 suffer for that. 22 Q. How would that process work? In terms of would the 23 Helpdesk gather names of post offices or was there some 24 other kind of process to notify those who were providing 25 the fixes of the affected post offices? 53 1 A. As far as I recall, what would happen is that this would 2 be a pattern developing and the pattern developing would 3 clang the bell of the SSC who would look at it and raise 4 the PinICL and either establish the workaround in 5 conjunction with development and apply it, apply the 6 fix, but I don't -- it's not something -- not everybody 7 dealt with these things society, so it was not something 8 that you would blanket apply. It's not a sticking 9 plaster for everybody. It's just for this specific 10 thing. 11 Q. You mentioned earlier that the third level of support 12 weren't great when it came to this like balancing or -- 13 A. Initially, they didn't have any experience of it. 14 I mean, all they had was what they had learned from 15 going on the course with -- they received the course 16 from Peritas and I think they may have even visited some 17 post offices. At one stage we had an adopt a post 18 office, so they would go through the process with them. 19 But they were not experts in balancing to the Post 20 Office's rules. 21 Q. Would they have the expertise to understand and spot 22 those kinds of trends that you've talked about? 23 A. Yes, because they had an impact. There was something -- 24 there was a signal that something had gone wrong and 25 that is something they can focus on and then find out 54 1 why and what needs to be done to correct it. 2 Q. Can we go to FUJ00079350. This is a "Live System 3 Performance Report" for February 2000 so, again, it's 4 after the rollout or after the rollout has resumed. You 5 are a recipient of this document. 6 A. Yes. 7 Q. Can we look at page 9, please. There are various issues 8 that are mentioned throughout this document. I'm going 9 to take you to them. Here we just have one, which says: 10 "Network -- Two periods very long calls have been 11 experienced on the ISDN network. Mitigating actions 12 have been put in place whilst the Riposte bug is 13 resolved." 14 A. Yes. 15 Q. Are you aware of what that's a reference to? 16 A. Not specifically. ISDN was not my favourite network 17 protocol. 18 Q. The reference to a Riposte bug there, were bugs with 19 Riposte common? 20 A. Oh, yes, as common as with any other software, yes. 21 Q. Were they more common with Riposte? Was there 22 a particular problem with Riposte? 23 A. I have to be a little guarded here, not because I wish 24 to conceal anything but because I wish to be fair to 25 other people's software. If I can't see the code, I'm 55 1 always upset. I can't -- I don't like not knowing 2 what's going on and when -- if it's in -- like with 3 Windows NT, the famous blue screen problem, we didn't 4 have access to Microsoft's code to go and fix it. 5 We didn't have access to this code to fix it. We 6 had to work through the reporting process, register 7 a fault, get Riposte to work it into their busy schedule 8 and wait for it to be tested, come back, test it again 9 and deploy it. That was always an element of delay that 10 doesn't help anybody. So ... 11 Q. Can we look at page 46 of the same document, please. If 12 we scroll down, it says: 13 "Riposte System Messages 14 "The number of messages generated by Riposte 15 functions eg: 16 "Log on/log off 17 "End of day reports 18 "Session transfers 19 "etc 20 "is significantly greater than the prediction 21 which was based on the CSR(NR2) Live Trial system. The 22 prediction was that 200 messages per counter per day 23 would be generated. Data from the live system indicates 24 that the number currently exceeds 500 per counter per 25 day." 56 1 Is this something you are able to assist us with 2 at all as to what that means? 3 A. I'm afraid not. I think you might be better to talk to 4 the development team on that. 5 Q. Scrolling down, "User Lock Requests". It says: 6 "CPs to remove unnecessary messages are being 7 raised starting with CP2253 which significantly reduces 8 the number of User_Lock_Requests generated by the 9 counter. This will both reduce the number of messages 10 in the message store and significantly reduce the load 11 on the Persistent Object Index ..." 12 I mean, this is all quite technical but is that 13 something you recall? 14 A. It sounds like -- I do recall something like this 15 when -- the "lock" is only requiring if you are going to 16 write. What's you're locking is the data from being 17 changed whilst it's in use but if you are reading it, 18 you needn't lock it. If it's locked because somebody 19 might be changing it then, fine, you have to wait for 20 the lock to be released. But I think there were -- 21 there was a criticism that they were locking everything 22 and that created too many requests. 23 Q. Riposte is mentioned there and we're also going to talk 24 about the EPOSS system. You laugh -- 25 A. No, I'm not laughing. 57 1 Q. Perhaps you can -- 2 A. I'm holding back a tear. 3 Q. I'd like to talk about your instinct on the mention of 4 EPOSS then. Can you tell us what was the reputation of 5 EPOSS in the office? 6 A. It's -- I think there are too many young people in the 7 room, actually. EPOSS was never an ideal system. 8 I'm sure it worked well in places where it was 9 designed for smaller numbers but I think it had -- we 10 had too many bugs with EPOSS. It just -- I can't 11 remember a time when EPOSS was the darling of the 12 family. It was always a problem. 13 Q. And -- 14 A. I mean, there was a time when we were thinking about 15 rewriting it completely but it's -- there was a system 16 that post offices used, which I can't remember the name, 17 but it was developed, I think, by an ex-postmaster -- 18 I'm sure somebody will know him. 19 Q. Is this the something Jackson? 20 A. Oh, that's it, Jackson. Now that seemed to work and had 21 the support of quite a lot of postmasters but we had 22 Riposte, and that was -- we had either to integrate with 23 Riposte or completely change. Now, if we completely 24 changed we'd change everything and I don't think that 25 was either in Post Office's interest or in our interest. 58 1 What we had to do was work with Escher and try and come 2 up with a Riposte solution by them, which met the 3 requirements and I'm not sure it was ever wholly 4 successful, and I'm pretty sure that the next generation 5 Horizon or Horizon Online changed that. 6 Q. How widely held was your view, that view that you just 7 expressed to us, of EPOSS effectively not being fit? 8 A. I didn't say that. 9 Q. No, well, that's why I used the word "effectively". 10 Please do -- 11 A. No -- hm. I can only measure it by "Does it make my 12 life simpler or more difficult", and it always made my 13 life more difficult. So I was never happy with it. It 14 was a very complicated system on which to -- we talked 15 a moment ago about reference data and reference data 16 being very specific about Post Office's products' price, 17 and so on, circumstances in which those things can be 18 traded. To build that in to something which was 19 designed for -- I think, An Post in Ireland used it but 20 were they anything like Royal Mail, you know, 21 Post Office Counters Limited? I don't think so. 22 So it was always adapting and it was possibly -- 23 I don't know whether we were big enough to warrant the 24 attention from Escher. 25 You know, they had tremendous ambitions around the 59 1 globe for this Post Office system. So I don't know. 2 Q. I will return shortly to EPOSS and some correspondence 3 between yourself and Terry Austin but, before I do that, 4 can we just look at FUJ00058190. This is the ICL 5 Pathway monthly report for February 2000 and it's 6 page 24 that I would like to look at. I am just going 7 to read to you that second bullet point under 8 "Acceptance Loose Ends", so if we could scroll down 9 slightly and just highlight that second bullet point. 10 It says: 11 "We have dealt with queries from POCL concerning 12 [Acceptance Incident] 376. One formal letter has been 13 responded to attempting to avoid the conclusion that we 14 had not found EPOSS reconciliation incidents that we 15 should have found or that we have not reported those we 16 did find. In reality CS are greatly hampered in 17 'spotting the incident' because the reports have not had 18 fixes implemented and report large amounts of do-nothing 19 information. We have attended the Release Management 20 Forum and proposed some reordering of the fix backlog, 21 but it will be at least until the first week of March 22 before this situation improves." 23 In Phase 2 we were told by Terry Austin that he 24 thought that you had written that particular paragraph 25 or at least had provided that content; is that right? 60 1 A. No, I think it was John Dicks. 2 Q. Is it something that you -- I mean, it refers there to 3 "CS", so customer service. 4 A. Yes. Well, CS would be the ones charged with spotting 5 the incident and if we couldn't spot it, then we'd be -- 6 we would certainly be hampered. 7 Q. Is it a concern that you recall or something that you're 8 able to assist us with? 9 A. No, I'm afraid not. Could I look at the rest of this 10 document? 11 Q. Yes, absolutely if we can -- 12 A. Where is it? Is it in -- 13 Q. If you would like the hard copy, it is your D18. 14 A. D18. 15 Q. It may be better -- we can come back to it at the end, 16 if that helps. 17 A. Just scroll on the screen would be fine. 18 Q. Where would you like: above, below? 19 A. Well, start from the top. I'll just have a look. Yes, 20 this is -- I could not possibly have written this 21 because this is written by John Dicks, it's "Customer 22 Requirements Monthly Report". Well, he wrote his own 23 reports. I wrote the customer service monthly report. 24 Q. Thank you very much for clarifying that. In relation to 25 the issue that it raises, is that something you 61 1 remember: issue spotting incidents? 2 A. Well, John was always very sympathetic to the problems 3 that we face. I mean, we were working, really, with one 4 hand tied behind our backs, really, because we -- we 5 can't see what's not reported and there could well be 6 problems, I'm sure there are problems, even today, that 7 have not been discovered yet. There are always bugs. 8 So I think he was being sympathetic to -- trying to stop 9 people saying "Well, customer service should have 10 spotted it" and, in fact, we probably couldn't have 11 spotted it but ... 12 Q. Is that because you were reliant on people calling the 13 Helpdesk to say "I've got a problem", or it's something 14 more than that? 15 A. No, I think it starts with that and then it's a question 16 of understanding in the system and if these were in 17 Riposte then it's over the Atlantic and trying to get 18 them to explain what went wrong. 19 Q. Can we look at FUJ00079333, please. This is the 20 correspondence between yourself and Terry Austin in 21 April 2000. 22 A. Yes. 23 Q. Perhaps we should start from the second page and if we 24 zoom out there's an email to yourself and others from 25 somebody called Pat Lywood. Could you tell us who Pat 62 1 Lywood was? 2 A. She's my Rottweiler. She was, I think, the epitome of 3 defending the product, defending the user, defending 4 customer service. She was wonderful at getting to the 5 bottom of problems. I remember going through a session 6 one evening when she said "I'm not going to be beaten by 7 this bleep, bleep piece of tin", and it was when we were 8 trying to get the -- it's the blue screen problem and 9 some other problems to do with the counter terminal 10 equipment. But, no, she was tenacious in her job. 11 Q. So she would identify for you -- 12 A. Yes. 13 Q. -- technical issues with Horizon? 14 A. Indeed, yes. 15 Q. Were you her line manager or did she report to you? 16 A. No, she reported in to the SSC and in to -- in to the 17 SSC. 18 Q. Who, other than yourself, would she routinely express 19 those kinds of concerns to? 20 A. Oh, to Mik Peach, to Peter Jeram, to Terry Austin, to 21 me. Pat would make sure that we knew when there was 22 a problem. 23 Q. This correspondence that you've recently seen is 24 concerning the CI4 implementation, which was an intended 25 improvement to the EPOSS system. Is that something that 63 1 you remember? 2 A. I don't remember the specifics of what was in it but 3 I do remember CI3, CI4. 4 Q. She says there: 5 "All, 6 "The following details were supplied by Phil 7 Hemmingway ..." 8 Do you remember Phil Hemmingway? 9 A. No. 10 Q. "... at a CI4 implementation meeting on 26th April. 11 This email details the current issues of which Phil is 12 aware." 13 She raises a number of issues. Towards the bottom 14 there we see "Performance issue", and then we see "Risk 15 of code regression". It says in relation to code 16 regression: 17 "There may be fixes that have been produced and 18 delivered into CI3 that have been missed from CI4. 19 "I will take this up with Dave Royle and ask for 20 assurance that all clone PinICLs have been tested", 21 et cetera. 22 A. Yes. 23 Q. If we go to the page before, there's an email from 24 yourself passing up concerns to Mike Coombs and Terry 25 Austin. If we look that bottom of that page, you say: 64 1 "Mike/Terry, 2 "Please see below, report from Pay Lywood on CI4 3 implementation. 4 "I am particularly concerned with the risks of 5 degraded counter and cash account performance and of 6 code regression between CI3 and CI4. Also, given the 7 dependence on [Post Office] Backfill Training but 8 without the benefit of the experience of PONU's track 9 record on this activity -- there must be significantly 10 increased risk that HSH performance against SLAs will be 11 severely impaired." 12 There are a few concerns you raise there. Can you 13 just take us through each one of those, please. 14 A. Okay. So CI3 to CI4. PinICLs that had been included in 15 CI3, if they had not gone forward to CI4, then we might 16 expect to have problems recur that we had thought we had 17 fixed and that is, you know, very bad. 18 The changes -- there were some changes, I believe, 19 to the cash account without the benefit of Post Office 20 Network's track record on this activity, Helpdesk 21 performance against SLAs will be impaired. Yes, I can't 22 remember what specifically they were but does it not say 23 over the page? 24 Q. Over the page, the original email? Yes. The 25 performance issues are slightly further down, if that 65 1 assists? 2 A. Wait for a moment. So we've got a new process 3 introduced to the cash account process, every office 4 will be required to declare non-value stock. The 5 backfill training had to be done by Post Office, 6 I believe. If they don't do it, then he won't be able 7 to balance or complete the cash account. 8 Length of time to do cash account was always 9 an issue. I mean, postmasters used to spend 10 an inordinate amount of time, late into the night, to 11 try to get the system to balance. 12 I think all this is saying, Mr Blake, is that 13 I was simply responding to my team's nervousness about 14 what had been produced for CI4 and that it wasn't what 15 we expected and we wanted to make sure that development 16 and the programme team knew about it. We weren't going 17 to just sit there and allow it to happen both to us and 18 to the Post Office. 19 Q. If we look at the page before and the bottom, your 20 email, the particular concern that you raise about code 21 regression, can you tell us a little bit more about 22 that? 23 A. Well, regression, there are two forms of regression. 24 One is that you introduce problems that weren't there in 25 the first place because you've made so many 66 1 modifications. Secondly, you don't include fixes that 2 you have already tried and applied to the earlier 3 release and they've been missed out. So, for instance, 4 if there's -- there's a first release and then there's 5 a maintenance release with some of these things in and 6 then, if the subsequent release -- real release, not 7 a maintenance release -- doesn't include the fixes that 8 maintenance release had, then you have regressed. So 9 that's what I mean by regression. 10 Q. Were you aware, I think you have mentioned something of 11 it in your evidence already, that in 1998 there was 12 a proposal -- an EPOSS PinICL Task Force, which raised 13 serious concerns about, for example, the code within the 14 EPOSS system? 15 A. Yes, I'm aware of it. 16 Q. Were you aware of it at this time? 17 A. Yes. 18 Q. If we look at Terry Austin's reply that is dated 10 May. 19 Can we scroll up slightly. Thank you very much. 20 I'll just read to you briefly from that. It says 21 at the beginning: 22 "Steve, I share your concerns regarding counter 23 performance and code regression." 24 He goes on in the next paragraph to say: 25 "I cannot give you a 100% guarantee that code 67 1 regression will not occur at CI4 because by its very 2 nature it is not fully automated and never will be. 3 However, our end to end processes are designed to reduce 4 the possibility of this occurring to an absolute minimum 5 and I have recently requested a reconciliation where it 6 is possible to do so. 7 "I also have no faith in PO backfill training ..." 8 Pausing there, can you just tell us what that 9 meant, the Post Office backfill training? I know you 10 briefly touched on it. 11 A. Post Office backfill training, I believe, was what we 12 gave to the Helpdesk staff and to the Post -- well, to 13 the Post Office staff to make them aware of how the 14 system would deal with new features or changed features. 15 So it was backfilling the training they'd already 16 received. So it's new stuff basically and ... I don't 17 think that Post Office were very good at getting that 18 across. 19 Q. Thank you. 20 Mr Austin's reply went to a number of people. One 21 of those was Gareth Jenkins. Is that somebody that you 22 were aware of? 23 A. No, Gareth was -- I think he was part of a Kidsgrove 24 group, sort of architect types. I may have met him but 25 I don't know him. 68 1 Q. Thank you. 2 Were you aware that on that same day, 10 May 2000, 3 Mr Austin sent a response which confirmed that -- it 4 recorded that a decision had been taken not to rewrite 5 the EPOSS system? 6 A. I'm not aware on which day that was, no. 7 Q. On the same day. Let me take you to that. WITN04600104 8 and it's page 10. So if we look at the response to you 9 about code regression, et cetera, was 10 May and then if 10 we look at page 10 of this document, so it's the page 11 before, there's an email there. I think you've seen the 12 evidence of Mr Holmes, haven't you, and I think this is 13 a document that was brought up on screen for Mr Holmes, 14 but if we can focus on the right-hand side, and -- 15 sorry? 16 A. Keeping his options open, isn't he? 17 Q. This concerns the rewrite of the EPOSS system and, 18 actually, it is the next page. It's the top of the next 19 page, which is the final -- that's page 9, yes, so if we 20 go over the page to page 10. So there we have 10 May, 21 so the same date that email was sent to you, and it 22 says: 23 "Following response received from [that's 24 Mr Coombs] ..." 25 A. Yes. 69 1 Q. "As discussed this should be closed. Effectively as 2 a management team we have accepted the ongoing cost of 3 maintenance rather than the cost of a rewrite. Rewrites 4 of the product will only be considered if we need to 5 reopen the code to introduce significant changes in 6 functionality. We will continue to monitor the code 7 quality (based on product defects) as we progress 8 through the final passes of testing and the introduction 9 of the modified CI4 codeset into live usage in the 10 network. PJ [that's Mr Jeram, I believe] can we make 11 sure this is [significantly] covered in our reviews of 12 the B&TC test cycles." 13 Do you remember this at all? Were you consulted 14 as to the closing of the recommendation to rewrite the 15 EPOSS system, rewrite and redesign the EPOSS system? 16 A. I remember the decision was that we were going to press 17 on regardless because the alternative was too expensive 18 and would have created huge delay in the programme, in 19 which case let me put my hand up to be the sacrificial 20 lamb and me and my team would have to battle through the 21 problems and cope with it. But I think Terry was 22 between a rock and a hard place there. He didn't -- he 23 really had a very, very difficult choice to make between 24 proceeding with what we had or starting from scratch 25 and, if you start from scratch, then. The further 70 1 ramifications of that change, I think, would be 2 unconscionable. 3 I mean, as a management team, this was the 4 accepted approach. 5 Q. The reference there to a management team, Terry Austin 6 was asked about who would make up the management team 7 and he gave your name as part of that management team. 8 Did you have a role in that particular decision? Do you 9 recall having a role in that particular decision? 10 A. No. 11 Q. Do you think you were part of the management team so 12 described in that correspondence? 13 A. Yes, I was part of the management team. We were -- the 14 managers were all part of the management team. As 15 I explained earlier, we had our blinkered view. It's 16 not entirely blinkered. We did see sideways a little 17 but we were focused on what our responsibilities were 18 and to try and do the best in that -- this was not -- we 19 didn't have the luxury of trying to create something in 20 advance of trying to sell it. This had been sold and 21 agreed, the Post Office had requirements, we had agreed 22 to meet those requirements and we had an issue here of 23 whether we stick with the product we had and try and 24 make it work or ditch it at great expense and start 25 again, and who knows what other consequences there might 71 1 be. 2 So even if I disagreed with it, and I probably 3 did, and you know I did from the point of view me 4 raising the issues earlier, the team decision was that 5 we had to proceed with EPOSS. I mean, they did change 6 the approach. It was called Rapid Application 7 Development, which was -- there weren't enough exponents 8 of that approach available to us to be able to make that 9 work properly I don't think. It's not my department but 10 it's what I remember. 11 Q. So is your recollection that Rapid Application 12 Development had a role in causing the problems with the 13 EPOSS system? 14 A. Well, I think it might have delayed the changes we made, 15 ultimately, I think, which was to go back to Riposte -- 16 to Escher and get them to make the changes. Rapid 17 Application Development was not new but it was a way of 18 doing things which required you to understand exactly 19 what you were trying to produce, whereas what we had was 20 a customer who decided what they wanted and we had to -- 21 we were much better off working from the specification 22 requirements and working through the whole process 23 formally. 24 But that takes a long time and, having started 25 where we started, I don't think there was any choice but 72 1 to proceed. 2 Q. Thank you. I'm going to move on slightly in time to 3 May 2000 and can we look at FUJ00003682, please. 4 These are minutes of the board of ICL Pathway and 5 you're there in attendance. Would you routinely attend 6 the board? 7 A. I attended a few. Particularly now you notice Mr Hirata 8 was there and Kurokawa San. These guys were looking at 9 investing a huge amount in ICL and I think there were 10 some flotation concerns at one point. So they were very 11 keen to make sure that their very -- a important 12 project, such as Post Office Counters Limited Horizon, 13 went well and they were very attentive. I was called 14 along to explain how well we were doing or how badly we 15 were doing. It's always the case. 16 Q. I mean, I said moving on in time but, actually, 9 May, 17 so this is just the day before the decision was taken 18 not to rewrite the EPOSS system, or certainly the day 19 before it was communicated to Mr Holmes and others. 20 A. I don't know the significance of that. 21 Q. No. Well, the significance is that that's not mentioned 22 in these board minutes but if we scroll down the focus 23 of the board minutes, or a focus, is on issues with the 24 Helpdesk rather than software issues. 25 A. Yes. 73 1 Q. It says there: 2 "Mr Stares reported the rollout was going very 3 well at more than 300 individual Post Office 4 implementations per week. The principal issues related 5 to Helpdesk and other service issues which were 6 addressed later in the meeting." 7 Do you remember that at all, the whole focus being 8 on the Helpdesk? Why was the focus on the Helpdesk at 9 that time? 10 A. Because -- I mentioned earlier penalties. Penalties 11 were hard-earned money that we had to give to -- well, 12 it was deducted from what we would get from Post Office. 13 Q. Perhaps if we turn over the page, then, this is the 14 presentation you give -- sorry, over the page to page 4. 15 It's the presentation that you give to the board on 16 service levels and it says: 17 "The principal issue was that service level 18 agreements were not being met and service activity, 19 particularly on Helpdesks was deteriorating." 20 There are mentions here of the red alert and 21 that's something you mentioned earlier in your evidence. 22 I'm happy to read you more from there but if you 23 remember, then I'm happy to go with your recollection or 24 allow you some time to just flick through that? 25 A. No, I presented the service performance statistics. We 74 1 in customer services had invested quite a time -- a lot 2 of time and effort in looking at the evidence from the 3 systems about how each individual transaction had 4 performed. We were counting how long -- you know, 5 counting the time, how long it took to answer calls to 6 fix calls, and so on. So we had quite a range of 7 service level performance statistics, all of which were 8 cash-related. If you don't achieve this minimal 9 acceptable level then you pay. I can't remember -- 10 there was another calculation for what we would pay and 11 I'm afraid I was firmly on the hook for that one. It 12 was my team that had to improve it, find ways of 13 improving that performance. 14 Q. What in particular was the problem at that stage? 15 A. Well, the Horizon System Helpdesk. I don't think we 16 had, as I said in my witness statement, the quality of 17 staff and management in the Helpdesk. They were also 18 remote as well. This is a bit of a lame excuse and 19 forgive me for that but they were not with us in the 20 same sense. I couldn't walk around the corner and ask 21 them what their problems were. I could go and I did 22 meet them sometimes in the evening when the problems 23 were there, but it felt remote. 24 I also felt that they were -- they could do 25 better. They were performing disappointingly and 75 1 I don't think their management team felt proud of what 2 they were doing either. It was just maybe the 3 complexity ty of the product, maybe the complexity of 4 the customer estate to get to grips with but we felt 5 seriously let down on two occasions, to the point where 6 I raised a red alert on ICL Operating Systems Division, 7 and that's very serious thing to do, because it brings 8 into marked focus the state of affairs in a particular 9 division. 10 It's up to that division to get itself out but it 11 puts a great deal of emphasis on every other division in 12 ICL -- it's called the ICL red alert system. It's 13 probably changed now. But it meant that all other 14 divisions had to offer whatever help they could to 15 recover the situation. 16 We were losing -- we were in danger of not just 17 losing money it was terminating the contracts. That was 18 issue. 19 Q. Where, in your view, looking back at it, was the 20 problem? Was it expertise? Was it experience? Was it 21 work commitment? Was it something else? 22 A. I don't know, but I think it would be unfair for me to 23 think that they could become experts. They were expert 24 in dealing with problems to do with PCs and printers, 25 and so on, but when it came to Post Office processes, 76 1 they are far more intricate and involved than I think we 2 could have expected them to have been familiar with, and 3 it look them some time. I think, ultimately, we lost 4 the contract. I can't remember. I'd gone by then but 5 I think it went to Atos. Somebody will help me there. 6 Q. In terms of burdens, was there too great a burden on the 7 Helpdesk? Were they receiving an unexpected level of 8 calls or calls that related to technical issues of 9 a significant or magnitude that wasn't expected? 10 A. Well, there were periods when -- actually, I think you 11 showed a graph earlier where the calls actually reduced 12 quite markedly. We put that down to the counter staff 13 becoming more familiar and experienced in using the 14 equipment and that's, I think, to be expected. 15 I think -- well, I go into my post office regularly and 16 they have no issues in operating it now. It's very 17 slick. 18 But how long did it take them to get to that stage 19 of comfort, confidence, ability to be able to perform 20 like that? I think we probably underestimated the time 21 it took them to come up to speed and we certainly 22 underestimated the time it took the Helpdesk staff to 23 come up to speed. 24 Q. I'm going to take you quite quickly through the issue of 25 reference data because you have already talked about it 77 1 but can we look at POL00028564, please. 2 This was a letter of 28 October 1999 from John 3 Meagher. The complaint there, I can summarise it, 4 I think it's perhaps the major concern that's mentioned 5 on the screen there. So there were some concerns about 6 the quality of reference data and it goes on to say: 7 "... they had a major concern with a fundamental 8 aspect of the reference data design and their ability to 9 support it." 10 Is this issue an issue that you recall, 11 October 1999? There's a reference, if we scroll down 12 the page -- sorry -- to yourself, a meeting with 13 yourself, Mike Coombs and John Dicks, and it says: 14 "... and noted the following in addition to the 15 concerns above: 16 "Pathway are concerned with POCL not maintaining 17 the agreed lead times between receipt of data ... and 18 activation of data ... 19 "Pathway are waiting for a Reference Data business 20 rules document ... 21 "... most significant issue for Pathway is that 22 the current design (agreed by all) for the provision of 23 data changes from the Post Office to Pathway results in 24 the delivery of large volumes of data which contain no 25 actual change for Pathway." 78 1 That's something you mentioned before about it. 2 A. I mentioned earlier, yes. Take bullet point number 1 3 with bullet point 3 for a moment. So the situation was 4 that Post Office would give us a huge amount of 5 reference data that we had to process and a lot of that 6 data was -- I don't like this word but it's nugatory. 7 It added nothing to the operation. It was simply saying 8 you know -- there weren't changes, they were basically 9 regurgitating the reference data and that meant a huge 10 amount of transmission of data across a pretty shaky 11 network, as it was. 12 When I say "shaky", it was ISDN. It was as good 13 as it could be at the time. It's not nearly as good as 14 you find now with modern internet connections. But, 15 yes, that was ... 16 Q. What impact would that have on the end user, so the 17 subpostmaster? 18 A. Well, if they didn't have the right reference data they 19 couldn't sell the product properly. If you update -- if 20 you have changed the price of a stamp from 10p to 20p, 21 all you've got is 10p in your reference data. You can't 22 sell those stamps anymore. 23 Q. Did the change of reference data or the volume of change 24 have any other impacts that you recall? So would, for 25 example, sending a lot over the ISDN line have other 79 1 impacts? 2 A. No, we had to do that anyway. But if they were to have 3 consolidated it and, if you like, contracted it, zipped 4 it, to the size where only changes were going out, then 5 that would have reduced markedly the volume of data we 6 had to transmit in the timescale, because there had to 7 be -- we had to have -- there was a lead time that we 8 had agreed between the receipt of the change and the 9 activation of the data. The activation means getting it 10 to the counter and saying "This is now your reference 11 data, so that you don't look at that table you look at 12 this table". 13 So, yes, quite a -- I mean, I did have a team of 14 people who were quite dedicated to handling reference 15 data and they worked mainly evening and night-type 16 shifts because that's when it all happened. 17 Q. Can we look at POL00028561. This is a letter from John 18 Dicks, which you're copied into, to John Meagher on 19 5 November 1999, and I'm just going to read to you the 20 very final paragraph. It says: 21 "This experience demonstrates ICL Pathway's 22 concern that the end-to-end reference data process is 23 not sufficiently robust." 24 Was that something that you shared -- a concern 25 that you shared at the time? 80 1 A. It wasn't sufficiently robust, it was distinctly shaky 2 and we needed improvements. 3 Q. Did it improve? 4 A. I think incrementally, yes, it did and it must have for 5 the system to have worked and to have been finally 6 accepted. But I don't know the timescales. 7 Q. If we look at POL00028552. There appears to have been 8 a reference data review in November 1999. If we go over 9 the page it says there -- it explains how the reference 10 data is provided and it expresses there Pathway's 11 concerns. It says: 12 "Recently, Pathway have raised concerns that 13 various aspects of the end-to-end reference data process 14 would appear not to be operating as efficiently as they 15 need to in order to support rollout and the ongoing live 16 operation." 17 Now, this period is before rollout. In terms of 18 ongoing live operation, do you recall ongoing concerns 19 about reference data? 20 A. I can't remember when concerns about reference data 21 subsided. So I can't be more explicit than that. I'm 22 sure that -- I'm sure there were changes in reference 23 data processes. 24 Q. Thank you. If that could be taken down I'm going to 25 move on to an entirely different topic, which is 81 1 involvement with prosecutions and investigations. 2 I think it was Jan Holmes' evidence that in 2001 3 your department, the customer services department, 4 became responsible for audit extractions. Is that 5 something you remember at all? 6 A. Yes. We were -- there was -- oh, is it a request for 7 information? There was an RFI, I think, process. Post 8 Office would ask us to produce an extraction of data 9 from -- I think, from a message store of the 10 correspondence servers and we would do that and package 11 it up and pass it on to them. 12 Q. Do you remember why it was your team that was tasked 13 with that? 14 A. There was no other team that could do it. I mean -- 15 well, development might have been able to do but -- no, 16 no, development couldn't. They didn't have access to 17 the live system. No, it was only the SSC team that 18 could do that, yes. We set up a special security 19 system -- let me think -- Jan Holmes actually commented 20 on it. Our security was improved. We had special card 21 access. Even I was not allowed onto the floor -- onto 22 the floor where these terminals were. 23 We had multiple -- you had to have two people 24 involved, so it was, I think, Mik, who was the SSC 25 manager. He wanted -- I think he said he wanted 82 1 a developer there as well as the SSC man, so that they 2 could together assure them what they had done was right. 3 Q. So who would typically be involved in that process of 4 data extraction for the Post Office? 5 A. I think -- well, the request for information would come 6 from the Post Office to Martyn Bennett's team. I'm 7 trying to think of our security manager. 8 Q. Graham Hooper, who was -- 9 A. Graham Hooper was one, yes. There was another guy. 10 Q. -- security function. Does that ring any bells? 11 A. I do remember Graham Hooper. There was another chap 12 who -- I can't remember his name but the security team 13 would receive the request for information and validate 14 it and it would be passed to Mik's team, not sure 15 whether -- it would be Mik probably, and he would assign 16 it to one of his team and somebody else, or to two of 17 his team, or one of his team and the developer, one of 18 the development team, would go and extract the 19 information. 20 Q. Were you aware that extracted information was being used 21 for the purpose of prosecutions? 22 A. Not while I was there. I did when I heard about this 23 I looked up -- one of you gentleman at the front will 24 remind me -- Tracy? Who was the first -- there was 25 a first lady in the time I was there that had been 83 1 prosecuted but I'm not aware that any request came 2 through to me or my team to provide that information. 3 SIR WYN WILLIAMS: It may have been Mrs Pamela Lock. That 4 happened in 2001. 5 A. Forgive me, I really don't know, but I do recall there 6 was one but I don't know what happened and I didn't know 7 that they were being prosecuted. But I did know about 8 a request for information. 9 MR BLAKE: Did you have any conversations with the Post 10 Office about their use of data for the purpose of 11 prosecutions? 12 A. No. 13 Q. Did you ever provide a witness statement for any 14 criminal prosecutions? 15 A. No. 16 Q. Were you involved in any way in identifying those who 17 were to give witness statements to support the 18 extraction? 19 A. Never, no. 20 Q. Thank you. I'm very briefly going to take you to 21 a document, FUJ00079790. This concerns the Business 22 Support Unit and the RED Audit. Can you tell us very 23 briefly who the Business Support Unit were. Is that 24 something you recall? 25 A. Yes, it was one of Paul Westfield's team. They 84 1 provided -- they were people who had provided 2 information to management, Post Office or Pathway, about 3 the business. They'd support the business and this 4 particular case is where we had to correct 5 a reconciliation exception database and the BSU operated 6 that. 7 Q. What was the purpose of the reconciliation database, the 8 RED? 9 A. I can't remember whether it was -- there was a stage 10 where we were responsible for -- when I say "we", 11 Pathway was responsible for some of the errors and had 12 to pay and I may be mistaken but I think it was to do 13 with finding exceptions in the data. 14 Q. If we look at page 4, the introduction there may assist. 15 I suppose my simple question is: did this unit have 16 anything to do with the prosecution of subpostmasters 17 relating to discrepancies, so far as you can remember? 18 A. I think absolutely not, no. No. This would be more 19 dealing with the Chesterfield team, I believe. 20 Q. What do you mean by that, sorry? 21 A. Well, where there were errors, reconciliation errors, 22 you had to trace the information back to source and 23 I think that was all part of what happened in 24 Chesterfield with the Post Office. 25 Q. Thank you. There's one final document that I would like 85 1 to take you to before, I believe, Mr Henry has some 2 questions and that is FUJ00001329. This is Pathway's 3 release policy. If we look at the abstract, it says: 4 "This document defines Pathway policy for the 5 identification and planning of new Releases of Software 6 and Data." 7 I believe this fell -- if we look at page 4, 8 I believe you were listed there as an approval authority 9 for this document. This release policy something you 10 remember at all? 11 A. No, it's not -- I'm sure -- I mean, it was -- in the 12 back of my mind, yes, it's there, but I can't remember 13 what went in it. This is what we did, when would we 14 make a release. 15 Q. Can I ask you to look at page 3, which concerns the 16 document history, so the way in which this policy was 17 drafted. There are various entries in this document 18 history that relate to comments being applied by Post 19 Office Counters Limited. If we look there at, say, 20 26 November 1996, 27 November 1996. If we look down to 21 the bottom, version 5.0, it says: 22 "Amended to reflect comments from Horizon/POCL 23 review." 24 To what extent do you recall the Post Office 25 having input into a document such as this, the Pathway 86 1 release policy? 2 A. Well, I think -- I seem to recall that Post Office had 3 to approve our releases. I mean, performing a release 4 is no mean thing. It has potential for disruption of 5 the whole estate. So there is a process to go through 6 to: number 1, prove we can put it out there; number 2, 7 prove that it works once it's got there and it would go 8 through a number of stages to assure both Post Office 9 and Pathway that we were not -- you know, it's do no 10 harm. So these things were quite important. 11 Q. So -- 12 A. It would cover things like when two types of release, 13 a major release and a maintenance release, and the 14 timing between them and the amount of testing and the 15 nature of the testing that would be done. Very, very 16 important. 17 Q. So that's the situation of sharing information with the 18 Post Office when it came to software releases? 19 A. Yes. 20 Q. We spoke earlier about the Known Error Log. When it 21 came to the minutiae of the detail of errors within the 22 system, that wasn't something that was interacted in 23 this kind of a way? 24 A. Not until -- I think the problem there was with PinICL, 25 we couldn't share the PinICLs with Post Office because 87 1 of data that was proprietary because I don't think it 2 was entirely -- well, it hasn't been -- it wasn't for 3 outside consumption. 4 But PEAK, I believe was. That was -- I wasn't 5 there when they did PEAK but I remember Mick and I think 6 Steve Parker and others were going to, were talking 7 about developing a system which could enable better 8 searching and allowing it to be available to the likes 9 of Post Office. 10 Q. So your recollection -- it's not something you were 11 aware of actually happening but your recollection before 12 you left was that they were moving towards a greater 13 sharing of those errors -- 14 A. Yes, yes. I mean, indeed, we shared the errors 15 verbatim, if you like -- what's the ... word-for-word -- 16 with them and you'll see as early as 1998, I think, the 17 CAPS board before Benefits Agency resigned from the 18 programme, there were PinICLs mentioned in there. And 19 wasn't it about PinICLs I was relating to Vince Gaskell 20 in my memo? 21 Q. So some of the content of some of the PinICLs would be 22 relayed to the Post Office but not the PinICLs 23 themselves? 24 A. Yes. 25 Q. In terms of the Known Error Log, I think your evidence 88 1 earlier was you're not aware of there being a formal 2 sharing mechanism, albeit they could have asked for it, 3 I think? 4 A. I believe they only had access when they were on-site 5 but they were not denied access, yes. 6 MR BLAKE: Thank you very much. I don't have any further 7 questions. There are some further questions. Sir, 8 shall we -- we potentially have time for a very short 9 break before or we can just continue. 10 SIR WYN WILLIAMS: I'm obviously anxious about the 11 transcriber, in particular. I think if the questioning 12 is to take longer than, say, about ten minutes we should 13 take a short break. So I'd like some guidance from the 14 questioners. 15 MR HENRY: Sir, I think if we do therefore take a short 16 break. 17 SIR WYN WILLIAMS: That's not to -- how long do you think 18 you'll be, Mr Henry? 19 MR HENRY: Well, I've been given quite some leeway but 20 Mr Blake has very helpfully addressed some matters which 21 means I can deal with them quicker, but I think I could 22 be between 20 and 30 minutes. 23 SIR WYN WILLIAMS: Then we had better have a short break. 24 MR BLAKE: Shall we say ten minutes? 25 SIR WYN WILLIAMS: Let's fix a time because then I'm -- 89 1 MR BLAKE: 3.40. 2 SIR WYN WILLIAMS: 3.40. Fine. 3 (3.34 pm) 4 (A short break) 5 (3.43 pm) 6 MR HENRY: Sorry, sir, I didn't see you were on the screen. 7 SIR WYN WILLIAMS: That's all right. I'm here. And I'm 8 glad you are speaking into the microphone. 9 MR HENRY: Rather too breathily, I fear. If I may be 10 permitted, sir, through you, to just thank Mr Muchow for 11 his openness, if that isn't inappropriate. I just 12 wanted to -- I have been given permission to go to 13 a number of documents but it may be easier if I just 14 deal with the subjects and only go to the documents if 15 I really do need to. Thank you. 16 SIR WYN WILLIAMS: Whatever suits you, Mr Henry. 17 MR HENRY: Thank you very much, sir. 18 Questioned by MR HENRY 19 MR HENRY: Sir, would you agree that although Mr Austin was 20 in a very difficult position, the decision to rewrite 21 inevitably led to subsequent code regressions and 22 repeated errors? 23 A. I'm not qualified to respond to that. I only know from 24 my own experience of software development, which is 25 a long time ago in another life, that these things are 90 1 always very challenging and that -- I don't believe he 2 was wrong but nor do I believe he was wholly right. 3 There was a balancing act to be had and the impact that 4 might have ensued on the estate probably would have had 5 an equally severe impact on the fortunes of ICL Pathway. 6 So I have to support Terry in the decision that he 7 championed. 8 Q. But I'm just -- I realise that you mustn't go outside 9 the boundaries of your expertise but just developing 10 what you actually told us when Mr Blake was asking you 11 questions that code regression, you said, was where 12 problems get introduced. 13 A. Yes. 14 Q. That is because there are purported solutions which 15 introduce problems. 16 A. I have never come across any coding situation where 17 changes to the code did not introduce new problems. 18 Q. Thank you. And also previous remedies, as you 19 mentioned -- although you didn't use those exact words 20 but I'm just trying to, as it were, tease it out -- 21 previous remedies were, let's say, overlooked. So 22 forgotten fixes or patches meant that the errors cropped 23 up again later on. 24 A. They may have done. I can't say that they did but what 25 my team pointed out was a concern -- this was one of the 91 1 last elements of Mr Blake's questioning. I was talking 2 about the CI3/CI4. Is that what you're referring to? 3 Q. Mmm. 4 A. Yes, I think there was a concern in my team that CI4 was 5 not exhibiting the remedies that had been applied to CI3 6 effectively. So whether they were identical I doubt, 7 because you've changed the mix of the software. The 8 recipe's changed. Whatever you do to it, you don't 9 expect -- unless you make exactly identical, then you 10 can't expect to get the same result. 11 So if the team had found errors in CI4 which they 12 thought had been cleared by fixes to CI3, then that 13 indicated that CI3 had missed some of those, and that's 14 all that we were trying to highlight. 15 Q. And you also mentioned that rapid application 16 development that there weren't enough staff. 17 A. As far as I remember, there was only one. 18 Q. Really. 19 A. I wasn't development director but it wasn't the sort of 20 development process that had been used before and so if 21 some bright spark said, "Well, why don't we tried to use 22 rapid application development?" it's very tempting to 23 say, "Ooh, that's a new idea. Let's try that. It might 24 give us something" and I don't think it did. 25 Q. Now, could I ask you, sir, about the phrase you used 92 1 about your team becoming a "sacrificial lamb". 2 A. Yes. 3 Q. Would that be because development were essentially 4 encountering all of these problems and not perhaps 5 addressing them satisfactorily and you were being left, 6 in your role from the point of view of customer services 7 and the Horizon Service Helpdesk, to sort of pick up the 8 tab? 9 A. No. No, I don't mean that at all. Those things should 10 have been resolved by testing. When the development 11 team makes the changes, they do unit tests, module 12 tests. They might even get as far as a -- they won't go 13 as far as a system test. They'll hand that over to the 14 test team who will put those things together piece by 15 piece and they will have a set of scripts which they 16 follow to try and prove that it's operating as per the 17 current specification. 18 When I said "sacrificial lamb" what I meant was 19 had there been any of those things left behind or 20 ignored or -- no, not ignored, overlooked, then my team 21 would have to cope with it. And there were never -- 22 there weren't penalties per se that directly resulted 23 from development. It was always the customer service 24 side. So that's what I'm talking about. I paid the 25 money -- 93 1 Q. Yes. 2 A. -- when we got it wrong. So that's all I was trying to 3 elaborate. 4 Q. So, in other words, forgive the biblical illusion but 5 the sins of development were visited upon the 6 Helpdesk -- 7 A. To all of us because we were not backward in coming 8 forward with our criticism of development. I mean, 9 there's a traditional friction, tension, love/hate 10 relationship between developers and supporters because 11 the support team always seems to get it in the neck when 12 things goes wrong and it's the developers who say, "Oh 13 well, they did it wrong". 14 In fact, it's all of the team that had to work 15 together to give the feedback back to the development 16 team to be able to make the changes and that's what Pat 17 was doing in that letter. She was saying, "This is what 18 we've found, we need to" -- and so it goes up the chain 19 of command and that's where I wrote to Terry. 20 Q. So without wanting to trivialise this in any way at all, 21 but were the difficulties that you were experiencing 22 with your department, as it were, having to pay the bill 23 but it was a little bit like whack-a-mole in that one 24 problem was apparently solved and then another one would 25 spring up? 94 1 A. No, that is trivialising it, with respect. The company 2 paid the bill. 3 Q. Yes. 4 A. Customer service calculated how much we had to pay. 5 Some of customer service was responsible for the 6 failures (for instance, the Helpdesk call to fix this 7 and so on) but we, as a team, would have to fix it. It 8 wasn't anything like whack-a-mole. Whack-a-mole would 9 be good if you could find all of the problems simply by 10 testing. 11 But what I mentioned earlier what I wished we'd 12 done was a little more hostile testing. So instead of 13 following a script, which is basically how we would like 14 the operative -- the operator of the terminal follows 15 a set of rules that they've been trained so to do. The 16 system is a program which is software, unintelligent. 17 It follows a set of rules that it's been trained to do. 18 If there's a mismatch, something has gone wrong and it 19 was always the difficulty in finding out what potential 20 things could go wrong that we hadn't envisaged. 21 So, for instance, a line dropping out in 22 mid-transaction. Well, if it happens at 10 milliseconds 23 into the transaction it will have one effect if it 24 happens 40 milliseconds into the transaction it may have 25 a completely different effect. Now how many problems 95 1 have I got? 2 So that was our -- so the sacrificial lamb is 3 really more to do with we were the ones sorting that 4 problems. 5 Q. I see. Thank you. 6 Arising from your answer that you've just given 7 about the operator having to perform a certain sequence 8 and the unintelligent program also having, as it were, 9 in tandem a certain sequence in response, would you be 10 able to help me because you talked about from 2001 that 11 your department was responsible for audit extraction. 12 A. Sorry, carry on then. 13 Q. Well, I want to know about ARQ data because you raise 14 this issue about providing information under RFIs. 15 Would the ARQ data not give the keystrokes that had 16 actually been entered by the operator? 17 A. I don't know, to be honest. When you say "ARQ" can 18 you ... ARQ. 19 Q. You weren't familiar with that term? Don't worry. If 20 you weren't, then I can move on. 21 A. It's evaporated from my ever-diminishing brain cells, 22 I'm afraid. 23 Q. Well, don't worry then. If you're not familiar with 24 that term, then I'll move on. 25 A. Is it somewhere documented in my bundle? 96 1 Q. No, it's simply from the point of view of the data that 2 would actually enable one to analyse what the operator 3 had actually done but if you're not familiar with that, 4 then I'm going to move on. 5 A. A log of the keystrokes? 6 Q. Yes. 7 A. I don't know what it was called but I think there was 8 a log of keystrokes, yes. 9 Q. Could I ask you please, though, arising from the word 10 "Tracy" that you mentioned, if I give her name as "Tracy 11 Felstead" does that help you -- 12 A. That name rings a bell, yes. 13 Q. It does ring a bell? 14 A. Yes, not from the point of view of being part of an 15 investigation but from what I read in one of -- because 16 I went back over the transcripts and testimonies of 17 postmasters who had been prosecuted during the time 18 I was there, and the only one I seem to remember was 19 Tracy Felstead, but I don't know why, and she wasn't 20 part of any -- I don't think we ever received a request 21 for information for her. 22 Q. So that isn't necessarily a recollection of what 23 happened in 2001, is it? You're just saying that the 24 name has become familiar to you -- 25 A. Yes. 97 1 Q. -- during the course of the Inquiry? Is that fair? 2 A. Yes. Yes, I wanted to see if I remembered a person to 3 sort of refresh my memory but, in fact, I don't believe 4 there was ever not one single request from Post Office 5 during my time there. 6 Q. Not one single request? 7 A. No, not that I was involved with, no. 8 Q. Can you help -- and I realise I'm not going now 9 specifically, since given the previous answer you have 10 given, but can you recall if there was ever any 11 discussion in respect of financial compensation for the 12 provision of such data in connection with prosecutions? 13 A. Yes, yes, yes. There was in one of the contract 14 meetings because this involved work which had to be 15 funded and so there was an agreement made that Post 16 Office would pay for each request. I don't know how 17 much it was but there was certainly a sum of money to be 18 paid. 19 Q. I appreciate that and thank you. I'm just trying now 20 to, as it were, probe a little bit further. Did it ever 21 come to your attention that perhaps somebody accused of, 22 let us say, fraud (whether it be theft or false 23 accounting) would have to pay Fujitsu for the provision 24 of information? 25 A. That's ludicrous. Sorry, no. No, no. Our customer was 98 1 Post Office Counters Limited. Post Office would be 2 required to pay to provide that. 3 Q. So that would have been, as you say, ludicrous and 4 completely irregular? 5 A. Yes. 6 Q. Sir, if you can help -- 7 A. Furthermore, we in Fujitsu ICL Pathway never had 8 dealings directly with postmasters, to my recollection. 9 Never did we speak to them about these things. 10 Q. Well, thank you, sir. 11 Could I ask you please -- you've just indicated 12 that you have reviewed certain of the proceedings, and 13 nothing wrong with that at all, quite properly. Did you 14 look at Mr Simpkins' evidence back in November? 15 A. Andrew Simpkins? 16 Q. Forgive me? 17 A. Was it Andrew? 18 Q. John Simpkins. 19 A. John Simpkins? 20 Q. Yes. 21 A. No. 22 Q. Well, I won't ask you any questions arising from his 23 evidence. 24 But could I ask you this: to your knowledge, were 25 there any tools or processes that you had to help level 99 1 3 staff, SSC staff, identify when there was 2 a distinction between a problem with the system itself 3 or whether it might have been user error? 4 A. I don't know. The SSC were a peculiarly clever team who 5 would work magic, I think, on looking at the system. 6 They had to be in order to find out the root cause. 7 There was no point in having a System Support Centre 8 that would simply say to development, "You've got a 9 problem, mate; sort it out". 10 What we had to do in the SSC was try and replicate 11 the fault and get to the point where we understood 12 exactly what the postmaster had done, what the system 13 had done, what the communications, the infrastructure, 14 the hardware had done, to generate that outcome and that 15 was something they were -- to my recollection, I'm very 16 proud of them. They did that very well. 17 Q. How familiar were you with, to use your expression, the 18 "magic" that they conjured up? 19 A. Now, now, that, sir, is - right -- 20 Q. Forgive me if that's loaded but -- 21 A. It seems to be magic to somebody who is not familiar 22 with the methods that they use but if you were to look 23 at somebody writing code today and somebody writing code 24 ten years ago, you'd understand. They were very, very 25 professional, very clever. Some of these people were 100 1 what we used to call dump crackers. They'd look at 2 binary bits, binary 1s and 0s and work out how things 3 had gone wrong in the program. But no. 4 Q. I think -- sir, I didn't mean magic pejoratively. 5 A. Good. 6 Q. But were you aware that SSC sometimes inserted or 7 injected, edited or deleted transactions to implement 8 fixes and that was done without the knowledge of the 9 subpostmaster in question? 10 A. Deleted transactions I don't think they did. I believe 11 what they were able to do -- with the NBSC's approval, 12 by the way, with Post Office's approval -- would be to 13 inject a balancing transaction. They couldn't modify 14 what had been done but if a postmaster had got into the 15 position where he couldn't rollover to the next cash 16 account period because of some problem and it could be 17 rectified by a balancing transaction -- so, for 18 instance, if he got something which said £900 and it 19 needed to be zero then, I think the SSC could inject 20 a minus £900 transaction. But that would be done with 21 the authority of Post Office. 22 Q. How closely involved were you in that process or were 23 you at one remove? 24 A. No, I was quite more than one removed. 25 Q. Substantially. 101 1 A. But I do know that -- I mean, I wandered around in the 2 evening sometimes to talk to them, and I think the 3 balancing transaction came up as an idea to help as 4 a workaround before a problem was fixed; so that it was 5 an idea that we put to Post Office that we could do but 6 I was never involved in actually doing it. 7 Q. But you weren't aware of the actual Post Office's 8 communication with the subpostmasters about that and 9 whether it took place or not? 10 A. I don't know if they did. I would have imagined that 11 Post Office -- I would have thought the NBSC (Network 12 Business Support Centre) would speak to the 13 subpostmaster and say, "This is what we've done; you can 14 now roll over onto the next period". 15 Q. I see. Were you directly involved in the removal of 16 that rolling over and the removal of the suspense 17 account yourself as a result of negotiations between the 18 Post Office and Fujitsu? 19 A. I really don't understand what you mean but I wasn't 20 but -- 21 Q. There came a point, sir, when a postmaster could no 22 longer, as it were, park disputes in a suspense account 23 and had to accept Horizon as being accurate before being 24 allowed to continue to trade on; in other words, the 25 right to question the data that Horizon had rendered, 102 1 the books that Horizon as it were had generated, was 2 removed. 3 A. It beggars belief that that actually is true but, if it 4 is, if that's what you say, it's true, then it's ... 5 Q. You weren't aware? 6 A. Not just not aware: I can't see why you would want to do 7 it. 8 Q. This is my last question to you, sir. You formed the 9 view that the negotiators on behalf of the Post Office 10 drove a hard bargain? 11 A. Yes. 12 Q. And they knew what they wanted, not necessarily from the 13 point of view of giving you a plan for the system in 14 advance, but they knew the commercial objectives they 15 wanted to achieve. 16 A. 100 per cent, yes. 17 Q. 100 per cent. And the priority was automation, reducing 18 costs at Chesterfield, manual balancing of books, 19 correct? 20 A. I think they are three of, yes. 21 Q. Thank you very much. 22 A. I think a lot more as well. 23 Q. Yes. 24 A. But that's it. 25 MR BLAKE: Thank you very much, Mr Muchow. Is there else 103 1 that you would like to add or clarify? 2 A. I thought about this before. Frankly, no. I think the 3 questioning today has given me ample opportunity to 4 express my opinion. 5 I'm very sorry that this Inquiry has to take place 6 at all. And, having listened to some of the 7 postmasters' testimonies, I'm quite distressed by it. 8 I never imagined that some company as respected as Post 9 Office could take, as Mr Henry just pointed out, away 10 the ability to challenge what is true. I mean, if you 11 know there's a problem, then you shouldn't have to sign 12 it away. So I was quite, quite -- quite surprised by 13 that statement, sir. 14 MR BLAKE: Thank you. 15 Chair, is there anything else that you would like 16 to ask? 17 SIR WYN WILLIAMS: No, I'm very grateful to you for coming 18 to give evidence and answering a great many questions 19 and also for your comprehensive witness statement. 20 Thank you indeed. 21 A. Thank you. 22 MR BLAKE: Thank you very much, sir. We will be back at 23 10.00 tomorrow. We have two witnesses: Mr Gilding and 24 Ms Parker. 25 SIR WYN WILLIAMS: Thank you very much. See you in the 104 1 morning. 2 (4.08 pm) 3 (Adjourned until 10.00 am the following day) 4 I N D E X 5 STEPHEN MUCHOW (affirmed) ........................1 6 Questioned by MR BLAKE ...........................1 7 Questioned by MR HENRY ..........................90 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 105