1 Wednesday, 17 January 2024 2 (10.02 am) 3 MR BEER: Good morning, sir, can you see and hear us? 4 SIR WYN WILLIAMS: Yes, thank you very much. 5 MR BEER: May I call John Simpkins, please. 6 SIR WYN WILLIAMS: Yes, of course. 7 JOHN GRAEME SIMPKINS (affirmed) 8 Questioned by MR BEER 9 MR BEER: Good morning, Mr Simpkins. My name is Jason Beer, 10 as you know, and I ask questions on behalf of the 11 Inquiry. Can you give us your full name, please? 12 A. John Graeme Simpkins. 13 Q. Thank you. You previously gave evidence to the Inquiry 14 on 9 November 2022. That was in Phase 2 of this Inquiry 15 and I think you were told on that occasion that there 16 was a possibility that you may be recalled in a later 17 phase or phases of the Inquiry. Thank you very much for 18 coming back again in this phase, Phase 4. 19 Since you gave evidence in November 2022 you have 20 made two further witnesses, I think, on 30 August 2023, 21 you made a 24-page witness statement with the URN 22 WITN04110200. Can you turn that up in the folder in 23 front of you, at tab A1. 24 A. Yes. 25 Q. If you go to the 24th page, you should see a signature? 1 1 A. Yes. 2 Q. Is that your signature? 3 A. It is. 4 Q. Are the contents of that witness statement true to the 5 best of your knowledge and belief? 6 A. They are. 7 Q. That statement, is this right, is principally about the 8 provision, the use and the reliability of ARQ data? 9 A. Yes, it's more about -- I was given an extract of an ARQ 10 data and could I discuss what it represents. 11 Q. Thank you. Then on 19 December 2023 you made a 10-page 12 witness statement with the URN WITN04110300, and if you 13 go to the tenth page of that, please, in tab A2, is that 14 your signature? 15 A. It is. 16 Q. Are the contents of that witness statement true to the 17 best of your knowledge and belief? 18 A. They are. 19 Q. That witness statement is principally about something 20 known as the Apex Corner incident; is that right? 21 A. That's correct. 22 Q. Something which you say you discovered between making 23 the second witness statement and the third witness 24 statement? 25 A. I was presented with another photocopy of an extract of 2 1 an ARQ -- sorry, of a report, and asked could I explain 2 this. 3 Q. Thank you. 4 Just by way of summary of your background, because 5 it's over a year since you last gave evidence, it is 6 right that you studied software engineering at the 7 University of Birmingham. 8 A. Correct. 9 Q. You're a member of the British Computer Society, 10 a chartered IT professional and an Incorporated 11 Engineer? 12 A. Correct. 13 Q. You joint ICL Pathway in July 1996 as an Application 14 Developer; is that right? 15 A. That is right. 16 Q. But shortly after then you moved away from development 17 work into a support role? 18 A. Correct. 19 Q. You worked in the predecessor department to the SSC, 20 third line support, during the period of the national 21 rollout of the Horizon system? 22 A. Yes, I did. 23 Q. So you were working there for Initial Go Live in 1996 24 and 1997? 25 A. Correct. 3 1 Q. You remain there for the course of the national rollout? 2 A. I did. 3 Q. You told us on the last occasion that your job title 4 then was Project Specialist? 5 A. Yes, I believe it's Product Specialist, actually. 6 Q. Okay, Product Specialist. Thank you. 7 Did you have a particular role at that time in 8 relation to the EPOSS software within the Horizon. 9 A. We supported it. 10 Q. And what did support of the EPOSS system consist of? 11 A. So if there were any reported incidents, live incidents, 12 mainly we were live for line support, so they would be 13 raised on a ticketing system and those tickets would be 14 passed to us to investigate the evidence. 15 Q. What would the investigation consist of and what would 16 you do in the course of the investigation? 17 A. Normally, you would investigate the -- well, it depends 18 upon the type of call, there may be events raised in the 19 data centre, there may be a -- evidence provided by 20 a subpostmaster or a user. There may be a -- another 21 feed of evidence from a database or some other source. 22 Then you would investigate the source of that evidence, 23 and you would probably gather evidence from multiple 24 locations, including the counter, some application logs 25 on the counter. You might look at the message store, 4 1 which was effectively the database on the counter. You 2 might look at the data centre, where you have harvesters 3 and other agents that worked with that data from the 4 counter, and the databases themselves. 5 Q. Would you please responsible for the development of any 6 fixes? 7 A. There was an idea we could look at doing workarounds, so 8 if a workaround was a -- either just telling the 9 subpostmaster how to work around a problem, or 10 potentially is there a workaround we can do, an example 11 might be clearing the print logs and things like that, 12 so we can actually clear a log and allow the 13 subpostmaster to continue working. But no software 14 fixes, we didn't produce, no. 15 Q. Who held the responsibility for software fixes at this 16 time, so in the Initial Go Live and then in rollout? 17 A. That would be the fourth line support team. 18 Q. The fourth line support team? 19 A. Correct. 20 Q. How would they be passed responsibility for writing 21 fixes? 22 A. So I mentioned a ticketing system, it was PinICL 23 originally, then PEAK. So we would add our evidence to 24 that system and then that ticket would be routed to the 25 appropriate team. 5 1 Q. Looking at that period as a whole, ie Initial Go Live 2 and then national rollout, what would your summary be of 3 the nature and extent of the problems with EPOSS? 4 A. There were problems with EPOSS definitely. It was a new 5 system, then -- I don't recall there being that many, 6 mainly because of the amount of staff the SSC had. 7 During -- 8 Q. Just to interrupt you there, you mean so that the 9 problems would be spread amongst that number of staff? 10 A. Yes and no. Sorry, what I meant was, initially, there 11 weren't that many staff in the SSC and we weren't 12 overrun with defects. Then, you're correct, as the SSC 13 grew, the defects were spread out but we did have 14 specialists in the team that concentrated on different 15 areas. Again, we weren't overrun. However, during 16 rollout itself there were a lot more calls than 17 post-rollout. 18 Q. Who was, if anyone, the EPOSS specialist? 19 A. I would say Anne -- 20 Q. So that's Anne Chambers? 21 A. Anne Chambers, Diane Rowe, Dave Seddon, Lina Kiang. 22 Q. Do you recall something called the EPOSS taskforce? 23 A. I don't, but I have seen documentation in this Inquiry. 24 Q. So only recently have you become aware of something 25 called the EPOSS taskforce? 6 1 A. Correct. 2 Q. So, at the time, you didn't know that there was a part 3 of Fujitsu given over to investigating a high number of 4 problems with EPOSS? 5 A. No, I was not. 6 Q. Were you aware at the time of a report that the EPOSS 7 taskforce wrote that recommended a rewrite of the EPOSS 8 software? 9 A. No, I was not. 10 Q. Therefore, I think it follows that you weren't aware of 11 the rejection of that proposal -- 12 A. Correct. 13 Q. -- and instead the adoption of a system of active 14 management, as it was called, of the EPOSS system? 15 A. Yes, that's correct and fixed forward is -- yes. 16 Q. You weren't aware of any of that going on? 17 A. No. 18 Q. I think you became a Team Leader in 2010 -- 19 A. Correct. 20 Q. -- reporting to the SSC manager. At that time was that 21 Steve Parker? 22 A. That's correct. 23 Q. You remain employed to date by Fujitsu as a Team Leader 24 in the SSC, the Software Support Centre? 25 A. Correct. 7 1 Q. Now, I want to ask you about the different species of 2 data held as part of the audit trail in Legacy Horizon 3 and Horizon Online. 4 A. Okay. 5 Q. So can we start, please, with Legacy Horizon. In fact, 6 can we turn to your third witness statement, please. 7 WITN04110300, and page 3, please. 8 Although this witness statement is to do with 9 something else, the issue that we mentioned at Apex 10 Corner, in paragraph 9 you set out a description of what 11 you call the life-cycle of a transaction in Legacy 12 Horizon. I just want to go through this because is what 13 you're doing here essentially setting out, stage by 14 stage, what happens when a transaction is undertaken in 15 Legacy Horizon? 16 A. Yes, I would expand it from that. It's all messages, 17 not just transactions. 18 Q. Thank you. So reading through it, you say: 19 "... messages (including transaction messages) were 20 written to the Riposte message store on the local 21 counter disk." 22 Can you explain for those who may not have listened 23 to all of the Phase 2 and 3 evidence what you mean by 24 that? 25 A. Riposte had distributed databases. Every counter had 8 1 its own message store, which was a non-sequel database, 2 effectively, and the messages that that counter used to 3 operate, including reference data and the transactions 4 it created at that counter, were all stored on the 5 message store which is a file, and that file is on the 6 local hard disk of that counter. 7 Q. So if there were five counters in a branch, there would 8 be five local counter disks; is that right? 9 A. Correct. 10 Q. If there were ten, there would be ten? 11 A. Correct, and if there was one, there was two counter 12 disks because that was a special case and had 13 a swappable disk. 14 Q. You continue: 15 "They were then replicated locally to other counters 16 within the branch or, in single-counter branches ([like] 17 Apex Corner), to internal removable mirror disks." 18 Can you explain what you mean by that, please? 19 A. Yes, so when a message is written it is broadcast 20 immediately to all local neighbours. Riposte has 21 an idea of neighbours and when you set up those five 22 counters in your example, you tell counter 1 about its 23 other four local neighbours, and counter 2 also about 24 its four local neighbours. 25 When you perform a transaction or any other message 9 1 on, say, counter 1, it will broadcast that to all 2 neighbouring counters, so that they will get a copy of 3 those messages. 4 As a single-counter disk, there was a single point 5 of failure on that counter, it had another version of 6 Riposte, effectively, installed on the counter as well. 7 Q. That's the mirror? 8 A. That's the mirror disk, which is a removable disk, and 9 the messages were again replicated to that. Also, the 10 counter node 1 was also called the Gateway Counter. 11 That had a remote access up to the data centre. So that 12 was the -- 13 Q. To the correspondence server? 14 A. Correct. So, in the data centre, we had correspondence 15 servers that that counter also replicated the messages 16 to. 17 Q. You continue: 18 "Legacy ... was primarily an offline system, so the 19 messages would be sent to the Correspondence Servers 20 periodically or immediately depending on the network 21 configuration. Every branch was assigned to one of four 22 'Clusters' ..." 23 That's clusters in the correspondence servers? 24 A. Correct, so we had 16 correspondence servers, so four 25 made up a cluster. So if that counter 1 replicated its 10 1 messages up, you had four copies of that message in the 2 data centre. 3 Q. You continue: 4 "... and this controlled which Correspondence Server 5 messages from that branch replicated to. There were 16 6 Correspondence Servers, and each one only contained 7 messages for a single Cluster. Once in the 8 Correspondence Servers, the Audit Harvester program 9 would copy all messages from the Correspondence Server 10 (ie a single Cluster) to a series of flat text files 11 labelled by Data Centre, Cluster and date." 12 Can you explain what the audit harvester program 13 was, essentially? 14 A. So there was an idea of agents which monitor messages 15 coming in to the correspondence servers. They 16 effectively listen at messages as they're inserted. 17 The audit harvester had a filter that basically got 18 every message as it came in and its job was to write it 19 to a flat file, so a basic file on the disk on the 20 correspondence server. 21 When it got to a certain size, it would switch and 22 start writing another file, and it began a new file each 23 day. 24 Q. When you were doing your work in the SSC, so when 25 a ticket came in, on PinICL to start with, which data in 11 1 that sequence of events that you've essentially 2 described, in that process that you've described, would 3 you seek to access? 4 A. Initially, the correspondence server because it's the 5 easiest to get to. 6 Q. Why was it the easiest to get to? 7 A. So when you were supporting a live data centre, you 8 had -- 9 Q. Sorry, the witness statement can come down. Thank you. 10 A. -- you had a computer dedicated to that network. It had 11 two-factor authentication and when you logged into it, 12 you would, from there, connect to the data centre, which 13 would allow you to connect to the correspondence servers 14 or databases. 15 If you wanted to go anywhere else then you would do 16 another hop, as it were, from the data centre. 17 Q. In your witness statement -- we're going to look at this 18 in a bit more detail in a moment -- you say that the 19 data that you found most valuable to access when 20 carrying out your work was in the message store? 21 A. Correct. 22 Q. Where are you referring to in the process you've just 23 described? 24 A. That is the correspondence server. 25 Q. In the correspondence server? 12 1 A. It should have all the messages from the counters. If 2 it doesn't match, effectively, with the call or what is 3 being reported, then you may go down and see if there's 4 a difference between the correspondence servers and the 5 counter. 6 Q. Indeed, that might be one of the very issues you're 7 investigating: a mismatch between what's held locally 8 and that which has made it to the correspondence server? 9 A. Potentially. 10 Q. So can you explain how, when you needed to access data 11 in the message store, you went about it in the SSC? 12 A. We did have some tools that we wrote, internally 13 ourselves, support tools. 14 Q. So you mean you wrote some code in order to get into the 15 message store? 16 A. You could do it multiple ways. One way is with the 17 tooling that we wrote ourselves. If you knew what you 18 were going to extract, Riposte allowed you to have 19 a query language, much like a SQL language. 20 Q. You have to explained what that means? 21 A. Sorry, structured query language, a database language, 22 where you could say, "I want this filled, this filled, 23 this filled, or attribute, from this counter in this 24 date range", and then -- so that, if you knew what you 25 were aiming for. If you didn't know what you were 13 1 aiming for, you would probably extract it all as text. 2 So the whole lot to one large text file and then start 3 filtering that through text editors. 4 Q. So Riposte had it own investigation/query system built 5 into it, a tool for extraction built into it? 6 A. It had a tool to extract, correct, yes. So you could 7 extract, actually the tool -- you could use part of that 8 structured query language as well. When you extract the 9 messages, normally you would just extract everything. 10 Q. Why was it necessary to write separate tools within the 11 SSC? 12 A. When you came across issues, you would learn to focus 13 your investigation and also some people didn't have the 14 skills to understand where -- write the language that 15 I was talking about, the Riposte grammar, so that if you 16 had a tool, everyone got it right all the time. 17 Q. Can you summarise, to your knowledge, the process of the 18 saving of, storage of and extraction of audit data? 19 A. So are we talking about audit data from Riposte into the 20 audit system, sorry? 21 Q. No, we're going to go on in a moment to speak about 22 something which has been described in the documents as 23 ARQ data. 24 A. Okay. I have a limited amount of knowledge about ARQ 25 data. 14 1 Q. As you say in your witness statement, it's not something 2 that was in your day-to-day use? 3 A. Yes, that's correct. But I can give you my 4 understanding of what have you. 5 Q. Yes. 6 A. So from those flat files we were talking about, which 7 are extracted by the audit harvesters, they are passed 8 to the audit system. The audit system then seals them. 9 So the audit system calculates a check digit on them, 10 and it puts that into a database and then that can be 11 reused later to make sure that it hasn't -- that file 12 hasn't changed whilst in audit. 13 Q. Stopping you there, what did you understand the purpose 14 of the retention of audit data to be for? 15 A. That's a good question. I presume it was such that when 16 messages that were no longer in the message store, or 17 messages that were no longer in the databases, or files 18 that we passed between us and third parties and third 19 parties to us, when they were no longer available on the 20 live system, we could go to audit and request for them. 21 Q. Did you understand, as the title of the data might 22 suggest, "audit data", that it was to be used for the 23 purposes of some sort of audit? 24 A. I didn't. I used it as an extension to history of the 25 data that's available to me. 15 1 Q. You mention that one reason that you understood it was 2 retained was that there was a limitation, a time limit, 3 on the retention of data in the message store. 4 A. Correct. 5 Q. How long was that limit? I think it changed. 6 A. It did, yes. I think I've -- I've definitely seen 7 message stores where it's 42 days. I think it also was 8 35 days, or something, at one stage. 9 Q. So did that mean data in the message store was not 10 available to you if you were conducting an enquiry, 11 an investigation, depending on the relevant time we're 12 looking at, more than 35 or more than 42 days after the 13 data had been created? 14 A. That's correct. Some messages do expire. There are 15 some messages that are effectively permanent and 16 objects -- you had objects and messages. Objects, 17 effectively, the last version of it was permanent and 18 never expired; but messages, other messages, did expire. 19 Q. Did you understand that when audit data, ARQ data, was 20 extracted by Fujitsu and presented to the Post Office, 21 it was presented in a filtered format? 22 A. Yes, I've seen some ARQ extracts that look like they are 23 filtered and then put in Excel. 24 Q. So the data has been manipulated from its original 25 source into a filtered format? 16 1 A. Correct. 2 Q. Was that something you were aware of at the time? 3 A. Not really, because we -- if we requested data from 4 audit, which I believe we did do, we got it back in the 5 basic Riposte -- 6 Q. Raw format? 7 A. Correct. 8 Q. Where did your understanding come from, that, for the 9 purposes of presentation to the Post Office, it had 10 been -- I've used the word "manipulated", that might 11 carry unwanted implications. Is there a technical term 12 for it? 13 A. Filtered? 14 Q. Filtered. 15 A. Sorry, that's just an off-the-cuff technical term. 16 I have seen examples of ARQs provided to me. 17 Q. Okay. Was software used to conduct that filtering? 18 A. Yes. 19 Q. What was that software called? 20 A. Again, I was presented with an ARQ and I think it had 21 a title on the top of the Excel spreadsheet which said 22 RQuery UK, which probably means Riposte Query UK, and on 23 the second tab of that Excel spreadsheet it had the 24 Flower language that was used, which is an XQuery 25 language, to say which fields to pull out and how to 17 1 filter it. 2 Q. Do you know who us wrote that software, that filtering 3 software? 4 A. No. 5 Q. Was that in-house again? 6 A. I expect it was in-house, as it said in-house on "RQuery 7 UK". I would have to talk to Gerald about that. 8 Q. So somebody within Fujitsu? 9 A. I would think so, yes. 10 Q. What was the purpose, to your understanding, of changing 11 the presentation of the data in this way or filtering it 12 in this way? 13 A. I do not know. I would expect it was to make it more 14 simple to understand. The original Riposte Attribute 15 Grammar is quite -- it's somewhere between XML and JSON 16 format. It's very structured in itself but not very 17 easy to read and there's lots of attributes in there 18 that probably won't make sense unless you have access to 19 the high-level designs. 20 Q. Thank you. Can we undertake a similar exercise to the 21 one that you undertook in paragraph 9 of your third 22 witness statement, "The life-cycle of a Legacy Horizon 23 transaction", for Horizon Online. You don't do this, 24 because that didn't arise in relation to the Apex Corner 25 incident, in your witness statements. 18 1 A. No, that's -- 2 Q. Can you broadly described the life-cycle of 3 a transaction in Online? 4 A. Yes, so when the transaction gets settled in Live, in 5 HNG-X, it's immediately broadcast up to the data centre, 6 to an OSR, which is an online service router, I think. 7 I might have to check that acronym but, if you have 10 8 OSRs, the messages are broadcast to them, then they will 9 then update the branch database or go via another route 10 such as CDP, which allows you to send messages out to 11 third parties. 12 So it depends on the type of transaction you were 13 doing but, if you were just doing a basic stamp sale, it 14 would go from the counter to the OSR and be recorded on 15 the branch database and then a response back to the 16 counter to say it was successful. 17 Q. Can I ask the same question, when you needed to access 18 data for the purposes of your work in the SSC, in 19 Horizon Online, which data would you access? 20 A. We would go to a database called the BRSS, the branch 21 support database. So the branch support database is, 22 very similar to the branch database, a live one, but it 23 has some replication software, it's Oracle, so it uses 24 GoldenGate, which copies the software -- sorry, all the 25 transactions that happen to the support database, and 19 1 the support database also keeps data for much longer 2 than is required in the actual live database. So we can 3 go back a year in the support database. 4 Q. Thank you. Can I turn briefly then to the role of the 5 SSC. Is it right that the SSC, the Service Support 6 Centre, worked closely with the fourth line of support, 7 Application Support Service, in the identification and 8 resolution of software incidents requiring bug fixes? 9 A. To an extent, yes. So they have no access to the live 10 information, so all evidence would be provided by us. 11 So, initially, we would do an investigation, gather the 12 evidence and then, if we can't explain it, then it will 13 probably go to the fourth line support team. If they 14 need any more evidence they would come back to us and 15 then, eventually, hopefully, they would be able to get 16 to the bottom of what the issue is. 17 Q. I want to look at a service description document from 18 2009, to see whether what it describes accords with the 19 position on the ground. Can we start, please, by 20 looking at FUJ00080066. Can you see from this page, 21 page 1, the title of the document is "Third Line Support 22 Service: Service Description"? 23 A. Yes. 24 Q. So this supposed to be a description of the service 25 provided by the SSC, yes? 20 1 A. Correct. 2 Q. We can see from the top right, the date is 4 September 3 2008 but, if we just go over the page to page 2, we can 4 see the document seems to have been approved only on 5 27 January 2009; can you see that? 6 A. Correct, yeah. 7 Q. If we go back to page 1, please. We can see the 8 originator of the document is Mik Peach. Was he the 9 manager at this time in January '09, so Mr Parker's 10 predecessor? 11 A. Correct, there was a manager in between them, yes. 12 Q. We can see over the page at page 2, about the middle of 13 the page, that a reviewer appears to have been Mr Parker 14 himself, yes? 15 A. Yes. 16 Q. So this document is, essentially, is this right, 17 a summary description of what you in the SSC were 18 mandated to do? 19 A. Yes. 20 Q. Can we go to page 14, please. This is under a big 21 heading on the previous page, we needn't look at it, 22 "Dependencies and interfaces with other operational 23 services", and there's a list of interfaces, so 24 interrelationships. It's paragraph 2.7.1.5, so the 25 second paragraph down on that page. If that just can be 21 1 enlarged, please -- thank you: 2 "The Application Support Service (fourth line) and 3 the Third Line Support Service work closely together in 4 the identification and resolution of Software Incidents 5 requiring bug fixes. If the scope of either the 6 Application Support Service (fourth line) or the Third 7 Line Support Service is changed, the completion of 8 Software Incident bug fixes would be the responsibility 9 of the remaining service." 10 What's that saying? 11 A. The first part is saying that -- 12 Q. Sorry, my mistake. What's the second part saying? 13 A. Oh, the second part? I have no idea. I'm presuming 14 it's either talking about merging third and fourth line, 15 or eliminating one of them, say fourth line would make 16 sense, so third line would also have to do bug fixes. 17 Q. I see, so it's talking about if there was a change to 18 the way that the support service was provided, that 19 either third line was extinguished or fourth line was 20 extinguished or changed, then the responsibility 21 described would vest in the remaining bit? 22 A. That's my reading, yes. 23 Q. Do you agree with the description of the interaction of 24 the third and fourth line support services? 25 A. Err ... yes. We also provided other things, other than 22 1 just software issues. We did lots of reporting. There 2 was facilities we provided other than just this but, 3 yes. 4 Q. To what extent was third line support involved in fixing 5 bugs? 6 A. We didn't actually do the fixes but we would help 7 identify the fixes, so we would provide the -- our 8 investigation, we would provide further evidence from 9 Live. I think that's probably it. 10 Q. So the SSC, would this be fair, should have good 11 visibility on the existence of bugs and the steps taken 12 to fix them? 13 A. We would have good visibility of bugs. Once the ticket 14 with all the evidence required is with fourth line, then 15 it may go off our visibility. In theory, we would have 16 probably created a knowledge article for that defect, so 17 that when another person gets a call they can identify 18 that that's already been identified and the call is 19 already with development. 20 Q. A knowledge article, is that different from a KEL? 21 A. Sorry, it's a KEL. 22 Q. Just explain what a KEL is to the uninitiated? 23 A. A KEL was Known Error Log. It's a repository of 24 knowledge articles that the first, second, third line 25 used. When we were investigating incidents, we could 23 1 search it with the symptoms that were provided to us 2 and, hopefully, find out that -- whether this incident 3 has been seen before, if there is a workaround, what 4 information do we need to gather if it's an ongoing 5 investigation? 6 Q. So, although it wasn't intended for this purpose, would 7 you agree that, if somebody asked the question from, 8 say, outside the organisation in 2005 "What known bugs 9 are there in the Horizon system and what steps have been 10 taken to correct them", the Known Error Log would be 11 a good place to start? 12 A. It's a good place to start, but it depends on the -- how 13 well that was house kept. So when the defect was 14 resolved and fixed, that needed to be fed back and -- 15 Q. Just stopping there, how well was it maintained? 16 A. I would say reasonably. I wouldn't say it was perfect, 17 I would say reasonably. 18 Q. Why wasn't it maintained more than reasonably well? 19 A. Because when the defect was closed, there was quite 20 often cloning of PEAKs and when a defect was closed it 21 may not be matched up to that KEL when it came back for 22 closure. 23 Q. You'll have to decode that, I'm afraid. I think 24 I understand what you mean but can you explain in 25 simpler language, please? 24 1 A. When a defect PEAK/PinICL goes to fourth line, they 2 could clone that ticket, especially if there is more 3 than one part that needs fixing, and when they have 4 released the fix, it may come back to us that that 5 PinICL or PEAK would come back to third line team for 6 closure because it was originated there, effectively, on 7 the PEAK/PinICL system, so when you do a final progress 8 it routes it back to the originator. 9 The pre-scanner in the SSC at that time would either 10 pass it back to the person who originally handled it to 11 make sure that is a reasonable closure or they may close 12 it themselves, and it's relied upon them to make sure 13 that they were aware of a knowledge article and update 14 it. 15 Q. So I think we agreed that if I was asking the question 16 in, say, 2005, of what known errors or bugs there were 17 in Horizon, the Known Error Log would be a very good 18 place to start? 19 A. It's a good place to start but you would need the 20 PinICLs or PEAKs to go with it. 21 Q. On what system was the Known Error Log kept? 22 A. It's the SSC's own corporate system, managed by us. 23 There were multiples throughout history where we managed 24 it ourselves and then effectively moved on to Fujitsu's 25 own internally managed services and then it was just 25 1 a virtual machine on that. 2 Q. Who had access, other than members of the SSC, to the 3 Known Error Log? 4 A. I believe the first line, second line, third line and 5 fourth line all had access to the Known Error Log -- all 6 Fujitsu staff, sorry. 7 Q. So it was a well known repository of information? 8 A. Correct. 9 Q. Indeed, that was its very purpose -- 10 A. Correct. 11 Q. -- that people knew about it and it's perhaps the first 12 thing one might reach to if a seemingly new problem 13 arose? 14 A. Correct. 15 Q. So they had electronic access to it, first, second and 16 fourth line support? 17 A. Yes. 18 Q. What about outside the support teams that you've just 19 listed; anyone else have access to it? 20 A. I don't believe so. 21 Q. In the period from 2000 to 2010, were you aware of any 22 challenges to the integrity of Horizon data being raised 23 by subpostmasters? 24 A. I was aware of any incidents raised during that time. 25 Q. I mean, that was your work, essentially, on a daily 26 1 basis? 2 A. Exactly. 3 Q. Should each of those have resulted in either a decision 4 to create a KEL, a Known Error Log, or to check whether 5 the issue being raised was adequately covered by 6 an existing KEL? 7 A. It should have been. 8 Q. Was that always done? 9 A. I believe so. You could also search the PEAK system to 10 see if there's any similar issues listed in PEAK. You 11 could search the first line Helpdesk system to see if 12 there's any similar issues there, as well as the KEL 13 system. 14 Q. Is it right that the SSC was not generally responsible 15 for reporting issues or the outcome of investigations or 16 the outcome of bug fixes back to the Post Office? 17 A. The ticket itself would be reported back. It had to, 18 I believe first line had to agree closure if the ticket 19 came through first line. But Service Management would 20 do that, while we were Incident Management, not Service 21 Management. 22 Q. So there was something called the Service Management 23 Team; is that right? 24 A. Correct. 25 Q. Was that also based in Bracknell? 27 1 A. Yes. 2 Q. So they were the point of contact back to the Post 3 Office; is that right? 4 A. Correct. 5 Q. Did they have access to KEL? 6 A. I can't remember. I imagine they did but I can't 7 remember, exactly, no. 8 Q. Did the Post Office have direct access to the Known 9 Error Log? 10 A. No. 11 Q. In your dealings with the Post Office, would you 12 understand that they knew of the existence of the Known 13 Error Log? 14 A. I don't know. I imagine we probably did refer to them 15 quite often. When we talked about an incident, we would 16 refer to a KEL reference, so that -- 17 Q. Why would you be referring, when you talked to the Post 18 Office, to a KEL reference? 19 A. I can't remember any instance of talking to the Post 20 Office but -- 21 Q. No, but, generally, why would you be talking about KELs? 22 A. Because it allows you to describe that there is a known 23 issue, we have referred it -- to it, this is, 24 effectively, a tracker of type for it. It has been 25 logged. 28 1 Q. You told us on the last occasion that Mr Peach, Steve 2 Parker's predecessor who left in 2009, introduced 3 something called the Service Management Portal or the 4 SMP, which was a website on to which was placed 5 reports -- 6 A. Correct. 7 Q. -- and that the Post Office had direct access to the 8 SMP? 9 A. Yes. 10 Q. With what frequency were the reports written? 11 A. You would probably have to ask Mik, however, I think it 12 was monthly reports but, presumably, he updated them 13 throughout the month and then published them. But 14 I can't be certain, I'm afraid. 15 Q. What were the monthly reports placed on to the Service 16 Management Portal about? 17 A. I believe they were about service impacting issues. 18 Q. What do you mean by that? 19 A. So any issues, any notable defects, any work that we had 20 done for the Post Office, any reports we had produced, 21 kind of metrics about what had happened in that month. 22 Q. So if there had been a bug, error or defect identified 23 and a fix applied to it, or some new code written to try 24 to correct the error, is that the kind of thing that 25 would be described in the monthly reports? 29 1 A. I expect so. Again, I would refer to Mr Peach though. 2 Q. Outside of that, the monthly reports on the Service 3 Management Portal, was there any formalised mechanism 4 for informing the Post Office about bugs, errors and 5 defects within the Horizon system? 6 A. I would expect that would be through the Service 7 Management Team. 8 Q. So that was the tool, was it? 9 A. Sorry, not the Service Management Portal, the Service 10 Management Team. 11 Q. Sorry, the Service Management Team. 12 A. Sorry. 13 Q. How many people worked in the Service Management Team? 14 A. I think about half a dozen. 15 Q. How did they get their information about what to tell 16 the Post Office? 17 A. Probably from the first line, third line. I'm not sure 18 where else. 19 Q. How physically would they get that information? 20 A. I know that Mr Peach provided a monthly report to the 21 Service Management Team. 22 Q. So the same thing, the thing from the Service Management 23 Portal, or a different species of report? 24 A. I don't know. I remember he -- mentioning he produced 25 a monthly report. 30 1 Q. Will you agree that there was a mechanism by which 2 Fujitsu told the Post Office what issues had arisen with 3 the Horizon system, how they had been detected, how 4 widespread the issue was, whether the issue affected 5 financial data and, in particular, balancing? 6 A. Yes, I believe that was the Service Management -- sorry, 7 Service Management Team's function. We definitely did 8 scoping when an incident happened to try to work out how 9 large an effect it has and who was affected. 10 Q. Ie whether it affected more than the one branch that 11 had, for example, reported the issue? 12 A. Correct. Once you know what marker that issue has, you 13 can search for it. 14 Q. Can I press you on how the Service Management Team got 15 its information from you in the SSC? 16 A. I would say that would be fed through our manager. 17 Q. By? 18 A. Through our manager, Mik Peach. 19 Q. How would your manager get the information? 20 A. He would get that from us. 21 Q. How would he get it from you? 22 A. Um -- 23 Q. You're working away in one corner of the room, 24 administering tickets, Anne Chambers is in another 25 corner of the room administering tickets, there are 31 1 another up to 25 people in the room administering 2 tickets, looking at your stack of tickets, processing 3 them, getting through all of the work. How was that all 4 of the information that you were creating, that you were 5 administering, translated to Mr Peach and then 6 Mr Parker, got over to the SMT and then got over to the 7 Post Office? 8 A. I believe it would have just been talking to him. He 9 sat in the centre of the office and we would tell him 10 what issues we've got if there's anything new. 11 Q. To your knowledge, did he, for example, regularly 12 periodically, say monthly, look at all of the PinICLs or 13 PEAKs that had been administered by the team and extract 14 from those the information that he judged it was 15 necessary for the Post Office to know about? 16 A. I don't know. I would have to ask him. I don't know 17 how he did his round robin of what the team has done 18 that month. 19 Q. You mention him sitting there. Presumably, he wasn't 20 there 24 hours a day -- 21 A. No, he wasn't. 22 Q. -- and I think there were shift arrangements; is that 23 right? 24 A. No. 25 Q. No. 32 1 A. So we worked from roughly 8.00 until 6.00. Core hours 2 were 9.00 until 5.30. There was an out-of-hours support 3 rota but that was just a team that worked normal hours, 4 and they would provide out of hours support as well, 5 passing a mobile phone round effectively. There was no 6 rota. 7 Q. So you would be sitting in your chair and he would be in 8 the room somewhere and you'd say, "Mik I've got a new 9 one here". What would happen then? 10 A. He would make a note of it, presumably, in his records. 11 I think he had database form for entering it directly 12 into the SMP. I don't know if that's there he kept his 13 records. But he definitely had a database form which he 14 would type it up on. 15 Q. He had access to the PEAKs and PinICLs himself? 16 A. Of course. 17 Q. So he could go back and check the ticket to see what had 18 been done or not done? 19 A. Correct. 20 Q. Did you ever see the reports that were passed to the SMT 21 or put on the Service Management Portal? 22 A. I have, yes. 23 Q. You have now. Did you at the time? 24 A. Yes. Quite often, if you wanted to get the details from 25 him, you sat next to him as he typed it up. 33 1 Q. So you could dictate or narrate what the issue was? 2 A. Correct. 3 Q. What did you understand the purpose of this 4 communication of information to the Post Office to be 5 for? 6 A. He was talking to Fujitsu at that time. He wasn't 7 talking to -- 8 Q. You mentioned that he wrote a monthly report that went 9 to the SMP -- 10 A. Oh, yes, correct. I don't know. 11 Q. -- which went to the Post Office? 12 A. Yes, for the SMP, I don't know. We presumably had some 13 agreement that he had to supply something or -- it was 14 very much off his own back, the SMP. I think he felt 15 that they needed some information and he went round 16 getting the server put in, and producing the software 17 for it. 18 Q. When did you first become aware that data produced by 19 the Horizon system was used for the purposes of criminal 20 investigations and criminal proceedings against 21 subpostmasters? 22 A. Anne Chambers was asked to provide evidence. 23 Q. So that would have been about 2006? 24 A. Yes, I think. 25 Q. Before then, ie from rollout until 2006, did you not 34 1 understand that the data was being used to investigate 2 criminally and then bring proceedings against 3 subpostmasters? 4 A. I don't believe so, no. 5 Q. The case in which Anne Chambers was involved was, in 6 fact, a civil case? 7 A. Right. 8 Q. Did you know that at the time? 9 A. No. 10 Q. Did you just understand it to be some form of legal 11 proceeding? 12 A. Correct. 13 Q. Can we go to page 2 of your witness statement, please, 14 your second witness statement, WITN04110200. Page 2, 15 paragraph 5. If we just blow up paragraph 5, please, 16 you say: 17 "... the SSC does not use and has never generally 18 used ARQ data in the course of its investigations. 19 Instead, for example in the context of Legacy Horizon, 20 the SSC referred to copies of the original Riposte 21 message store for the relevant branch when investigating 22 and diagnosing potential issues with the system. In 23 this regard, the raw message store contained information 24 additional to that in the filtered ARQ spreadsheets, and 25 provided a much more comprehensive account of the data 35 1 held in the audit archive." 2 So the SSC did not generally use ARQ data but used 3 a message store. Was that because there was more data 4 held in the message store beyond that which was produced 5 as a result of a filtered ARQ request? 6 A. Yes. 7 Q. What extra information was available in the message 8 store, as opposed to the audit archive? 9 A. I'm differentiating, I think, between ARQ here and 10 I think the raw is held in the audit, but the ARQ is 11 filtered. 12 Q. That's not precisely what you say here, is it? You say: 13 "... the raw message store contained information 14 additional to that in the filtered ARQ spreadsheets ..." 15 My question is: what additional? 16 A. Sorry, I was trying to -- yeah, okay. So there is more 17 data in the raw Riposte message store. However, I do 18 believe the raw message store is available from audit. 19 The ARQs I've seen are filtered and only put out certain 20 fields. 21 Q. Okay, so -- got it. So there's three things we're 22 talking about. 23 A. Yeah. 24 Q. Message store number 1, filtered ARQ data number 2, and 25 ARQ audit archive, number 3? 36 1 A. No. 2 Q. Okay. 3 A. I think I'm just talking about two things: filtered ARQ 4 and raw message store. So the reason you would go to 5 audit is if it's been archived off and you can get the 6 raw Message Store. The ARQs I've seen aren't -- are -- 7 because they are filtered are missing a lot of relevant 8 messages would be looking at. 9 Q. My question is, what a lot of relevant messages are they 10 missing that you would be looking for? 11 A. Okay, such as reference data. So reference data 12 controls how the counter operates. 13 Q. So just explain to us -- many of us know but for those 14 that don't -- what reference data is, please? 15 A. So reference data is configuration information for how 16 the counter operates, what it can sell, how much it will 17 sell it at, what buttons and configuration is available 18 to it. When you do some transactions, how -- what are 19 the steps of those transactions take? 20 Q. Thank you. So it would be missing reference data? 21 A. Those ARQs just seem to be events and transactions that 22 I've seen so far. 23 Q. Okay, you were in the middle, I think, of providing us 24 a list of things that were missing. 25 A. Yes, there would be additional attributes that aren't in 37 1 those ARQs I've seen. 2 Q. Such as? 3 A. Such as the NUM. So each message is written with the 4 group ID, which is the branch, node ID, which is the 5 counter position, and NUM, which is a unique 6 incrementing counter. That allows you to see exactly 7 what messages have been produced and you won't miss any, 8 and gives you the order that they were committed to the 9 message store. 10 There will be other attributes such as if you were 11 doing a banking transaction, you have a request, 12 authorisation, confirmation, handshake between the data 13 centre -- 14 Q. Just explain what a handshake is? 15 A. So when you start doing a banking transaction, you would 16 write a request message in at the message store. That 17 gets transmitted to the data centre, picked up by 18 an agent. The agent goes to the banking engine, sends 19 it on to the financial institute. Get it back with 20 an authorisation, which goes back down through the 21 agents, back down to the counter, the counter says, 22 "Okay, that's been authorised", and then you confirm it 23 at the counter. 24 That gets harvested back up to the data centre and 25 then we would reconcile that. So the handshake is the 38 1 passing of the messages backwards and forwards. 2 Q. Of course you've listed two species of data that are 3 missing from filtered ARQ that you could see in the 4 message store. Is there a third? 5 A. I think there was many. I'm struggling to recall 6 different types but almost anything that it -- AP 7 transaction -- 8 Q. Explain what AP transactions are? 9 A. Automated payments. Automated payments are like your 10 bill payments, BT payments, things like that. Again, 11 the system would write recovery data when you're halfway 12 through, until you've completed, so that, if it failed, 13 it would take that recovery data and ask you about that 14 transaction that was partially completed. 15 Q. Does all of this explain why you would go to the message 16 store and not to filtered ARQ? 17 A. Yes, because you see the whole picture. 18 Q. Would you agree that it's unwise to seek to base 19 conclusions on the basis of the filtered ARQ data, in 20 particular as to the health and integrity of the data 21 that Horizon has produced? 22 A. The health you could not decide from those ARQs. The 23 integrity of the transactions you may be able to, if 24 you've got the physical paper copies as well in the 25 branch: you could do a comparison between what the 39 1 system has and the branch has. 2 Q. If we go forward to paragraph 12 of your witness 3 statement, please, that's on page 4. You're here 4 referring to where you refer to the ARQ spreadsheet, 5 that's a spreadsheet you were asked to analyse -- I'm 6 not going to ask you any questions about it -- in 7 relation to Mr Lee Castleton and some days of ARQ data 8 at the Marine Drive branch? 9 A. Correct. 10 Q. You say: 11 "... if I refer to that ARQ spreadsheet by way of 12 an example, my view is that the data provided in the ARQ 13 spreadsheet does not contain sufficient information for 14 a postmaster to assess the healthcare of the Horizon 15 system at their branch. The ARQ spreadsheet shows only 16 those transactions recorded by the system. It shows 17 there were no receipts and payments mismatch within 18 those transactions and that there were no system 19 [faults] that required recovery. However, it does not 20 demonstrate the health of the system beyond those 21 parameters." 22 You say in that paragraph "The ARQ spreadsheet shows 23 only those transactions recorded by the system"; can you 24 see that, the fourth line, second sentence, yeah? 25 A. Yes. 40 1 Q. What did you mean by that, "The ARQ spreadsheet shows 2 only those transactions recorded by the system"? 3 A. The ARQ that was presented was a filtered subset of just 4 the transactions. 5 Q. Are you, by that sentence, also stating that the 6 additional message store data that you have referred us 7 to today may assist in showing the existence or the 8 conduct of transactions as between the local counter and 9 the centre, that are missing from the ARQ spreadsheet? 10 A. If you were to have failed banking transactions, for 11 example, or an AP transaction that's still yet to be 12 recovered, then I would agree. 13 Q. They wouldn't show up? 14 A. I'm just trying to think. The banking one, whether that 15 would show up as a zero value failed transaction or not. 16 It may still show up that there was a nil banking 17 transaction, but if the AP one was not completed then 18 I don't believe that would show up. 19 Q. To take an example, if we go to your third witness 20 statement, please -- sorry, your second witness 21 statement -- at page 17, paragraph 34, if that can be 22 blown up, please. You're here addressing an issue that 23 the Inquiry ask you about, which concurrent or 24 simultaneous logins, yes? 25 A. Mm-hm. 41 1 Q. You say: 2 "Although there have been issues with concurrent 3 logins ... an initial observation is that the ARQ 4 spreadsheet [that's the same one we're talking about] 5 for this instance does not appear to contain evidence 6 that a user was logged on to two counters 7 simultaneously." 8 I'll miss the next bit out. Then you say: 9 "In order to determine more conclusively what 10 happened at the branch, access to the raw message store 11 would be required." 12 Does that paragraph there and what you tell us in 13 it, reflect the fact that you as an expert in the 14 operation of the system or a person with expertise in 15 the operation of the system, would not be prepared to 16 draw a conclusion on the ARQ data alone. 17 A. Yes, it does. Because I am talking about a session 18 transfer, and a session transfer writes multiple 19 messages as it takes the transactions from one counter, 20 puts them in a blob attached to a message and then 21 transfers it other counter, and you can clearly see that 22 in the message store. 23 Q. So you wouldn't be prepared to draw conclusions without 24 access to the raw message store and would you say that 25 it would be wrong to ask other people to draw 42 1 conclusions on the basis of just the data that appears 2 on the filtered ARQ spreadsheets? 3 A. That point in 34 that I'm talking about, I would about 4 99 per cent sure that is what's happened from the 5 evidence between the events and the transactions. Your 6 question is very much wider but I would say, yes. 7 Q. Thank you. That can come down. 8 Was the known or was the limitation in ARQ data 9 widely known or recognised within the SSC, ie the 10 limitations of the ARQ data that you have mentioned to 11 us today, was that known within the SSC widely? 12 A. Not really. If we requested the information from the 13 audit, we would have got it in the raw format. We 14 wouldn't have had it in those Excel spreadsheet formats. 15 Q. Did you know at the time that what was being presented 16 to the Post Office and then used in court was the type 17 of filtered ARQ data that you have now seen in the case 18 of Mr Castleton? 19 A. I don't believe I saw that, no. 20 Q. Forget his case. 21 A. Yeah, sorry. 22 Q. Individually, I'm using that as an example. 23 A. I don't believe so. We did use to get PEAKs passed to 24 us with events -- counter and data centre events on to 25 filter and say do any of these events have any impact 43 1 upon a branch? They were in Excel spreadsheets but, 2 again, they looked like a complete extract from the 3 Tivoli database for the events but, no, I don't recall 4 any ARQ in those format. 5 Q. For example, in your experience, would Ms Chambers, Anne 6 Chambers, have been aware that there was substantially 7 more data available in the message store than was 8 provided in a standard ARQ package? 9 A. I'm sure Ms Chambers would have gone to the raw data, as 10 well, to do any analysis, yes. 11 Q. That is a different question, where she would have gone. 12 She would have done, I think, the same as you and gone 13 to the raw data. I'm asking whether you think others, 14 including her, were aware that the data being presented 15 to the Post Office in the filtered ARQ format contained 16 substantially less data than was available? 17 A. I'm sure if she saw that ARQ spreadsheet, she would have 18 known and, if any of the SSC saw that, they would have 19 known, but I wasn't aware of what the ARQs looked like. 20 Q. Were you or, to your knowledge, any of your colleagues 21 in the SSC ever asked to provide Fujitsu advice on the 22 range of data that was available and which, therefore, 23 ought to be presented for the purposes of civil or 24 criminal investigations? 25 A. No. 44 1 Q. Was that ever a matter of discussion, so far as you were 2 aware? 3 A. No. 4 Q. Were you aware of a branch within Fujitsu called 5 Litigation Support? 6 A. From yesterday, yes. 7 Q. You only learnt that yesterday? 8 A. Correct. 9 Q. Does it follow that Litigation Support, the people that 10 were providing the ARQ data to the Post Office, never 11 spoke to you or, to your knowledge, anyone within the 12 SSC about the range of data that was available, 13 additional to that which they were sending over in the 14 ARQs? 15 A. I would have expected, if they were concerned -- well, 16 I would have expected that that -- these ARQs had been 17 designed by someone, they would probably have been 18 architects and they -- I am presuming that they have 19 been agreed with the Post Office. That should have been 20 an architect-level discussion about what is available 21 and what should be provided. I don't know if that's 22 a Litigation Team level. I would have thought they 23 would just provide what has been designed in the system 24 for them. 25 Q. Going back to paragraph 12 of your witness statement, 45 1 please, that's on page 4 -- that can be blown up, thank 2 you -- you say, second line: 3 "... the ARQ spreadsheet does not contain sufficient 4 information for a party to assess the health of the 5 Horizon system at their branch." 6 Then the last line: 7 "... it does not demonstrate the health of the 8 system beyond those parameters." 9 What do you mean by the "health of the system"? 10 A. I would expect events, so Windows events of the counter 11 itself. I would expect events from the data centre, 12 mainly the harvesters, to say if there was any issue 13 harvesting the data written by the branch. I would have 14 thought about the logs that were written at the data 15 centre -- sorry, at the counter, audit PS standard logs. 16 There are reports generated at the data centre where 17 it's checking the transactions as they're entered into 18 the databases, for receipts and payments and they 19 regenerate cash accounts, those reports. I would have 20 thought about the tickets raised, if there were any, 21 PEAK and as well as TfS, so if our own internal systems 22 picked up, for example, any issues, they may be raised 23 as a PEAK/PinICL, as well as the TfS raised ones. 24 Then, going back to the Riposte, I think I detail in 25 here I would have taken from the balance messages 46 1 written in the Riposte system, so you -- when you're 2 calculating creating the current balance of, for 3 example, cash, you would take what was the opening 4 position for your current cash account, you would add up 5 all the transactions for your current cash account and 6 then you compare that to the declaration that the 7 subpostmaster enters and then you will see if there's 8 any discrepancy. 9 As the subpostmaster is doing the overnight cash 10 holding every night, you should be able to quickly see 11 if there is a diversion between the system generated 12 figure and the subpostmaster's entered figure and so 13 that would be the point when you start investigating. 14 Q. Was the first time that you saw a filtered ARQ 15 spreadsheet provided to a subpostmaster when we, the 16 Inquiry, showed it to you for the purposes of this 17 Inquiry? 18 A. I believe so, because that's the first time I noticed 19 the RQuery UK and the Flower language because, in my 20 second witness statement, I think I had some trouble 21 working out whether a time was the start time of 22 a transaction or the time it was committed, and I worked 23 out it has to be the start time. But, by seeing the 24 spreadsheet presented to me in the third witness 25 statement, it actually has the filter there, you can see 47 1 it's the start time. So that would have helped me with 2 my second witness statement. 3 Q. So it was only in 2023 that you saw the type of data and 4 the extent of the data and how it was being presented 5 that was being transmitted from Fujitsu over to the Post 6 Office for the purposes of criminal proceedings? 7 A. I believe so, yes. 8 Q. To your knowledge, did anyone in Fujitsu ever explain 9 the limitations of the data that was being provided to 10 the Post Office? 11 A. No. 12 MR BEER: Sir, that would be an appropriate moment to take 13 a break in the topics that we're addressing. 14 SIR WYN WILLIAMS: Before we do, can I just ask 15 Mr Simpkins -- this is just to check that I haven't 16 misunderstood earlier evidence by other witnesses, 17 Mr Simpkins, so if you can't answer, that's not 18 a problem -- but when we heard extensive evidence from 19 Mrs Chambers, she told us two things: essentially, she 20 was unhappy with her experience in giving evidence in 21 the Lee Castleton case; and, secondly, that she'd 22 written quite a detailed memo about her experiences and 23 what she thought ought to happen as a result of it. 24 My recollection is that, after that, nobody in third 25 line support actually did give evidence in either civil 48 1 or criminal proceedings. Have I got that right, as far 2 as you're concerned? 3 A. Yes. 4 SIR WYN WILLIAMS: Fine. So it follows that, to this day, 5 and you're still there, as you've told us, no one from 6 third line support has given evidence in a criminal or 7 civil trial and, as far as you're aware, no one in third 8 line support has made a witness statement; is that 9 correct? 10 A. Only to the Inquiry, yes. 11 SIR WYN WILLIAMS: Yes, sure. I meant a witness statement 12 in civil or criminal proceedings. 13 A. Correct. 14 SIR WYN WILLIAMS: Fine. Then this is a long shot: when 15 Mr Beer was asking you questions about the SMT 16 disseminating material to the Post Office, he used the 17 expression "the Post Office". Do you happen to know 18 which department of the Post Office that sort of 19 information might have been disseminated? 20 A. I don't, I'm afraid. I even added users into that 21 system. I remember doing that, adding their logins, but 22 I have no recollection of who it was or what parts of -- 23 SIR WYN WILLIAMS: All right. Thank you very much. 24 What time shall we start again, Mr Beer? 25 MR BEER: 11.35, please, sir. 49 1 SIR WYN WILLIAMS: Fine. 2 MR BEER: Thank you. 3 (11.20 am) 4 (A short break) 5 (11.35 am) 6 MR BEER: Good morning, sir, can you continue to see and 7 hear us? 8 SIR WYN WILLIAMS: Yes, I can, thank you. 9 MR BEER: Can we turn up page 4 of your witness statement, 10 please, your second witness statement, and look at 11 paragraph 14 at the bottom. You say, if that can be 12 expanded, thank you: 13 "Beyond the data described above, it would also have 14 been useful for the postmaster to have visibility of (i) 15 the opening figures from the last rollover, (ii) 16 a running total of the sales, and (iii) the daily cash 17 and stamp declarations made by the postmaster. Access 18 to these records would have allowed the postmaster to 19 compare the Horizon generated figures against the 20 declarations made by the postmaster from the point of 21 the last rollover. A comparison of these figures would 22 show the point at which the two figures diverged, 23 allowing the postmaster then to check what was happening 24 at the branch at that point in time." 25 Is it right that the three species of data that you 50 1 mention there are not shown on the filtered ARQ data? 2 A. So the opening figures are not, the declarations are 3 not, however, the transactions are. 4 Q. So (i) no; (ii) no; but (iii) yes? 5 A. No, sorry: (i), no; (ii) you have the sales in those 6 ARQs -- is yes; (iii), no. 7 Q. As far as you were aware, was there any facility for 8 a subpostmaster in branch to either run reports on 9 Horizon which would generate that information, in 10 categories (i) and (iii), or otherwise to keep track of 11 that information by some other means? 12 A. Yes, the -- one would be from the stock unit rollover, 13 would detail the opening figures. 14 Q. How would the subpostmaster obtain that information in 15 branch? 16 A. When they do the stock unit rollover, the printout on 17 that will display the opening figures. 18 Q. So they could access their print -- 19 A. Correct. 20 Q. -- from the previous rollover? 21 A. Yes. The -- there was sales transaction reports 22 available in branch. You would enter in a list of 23 parameters to the query report to say which stock unit, 24 which start date, which end date, things like that -- 25 I think you could input what product, I can't remember 51 1 exactly -- and the events. So the declarations would be 2 shown as events and you did have event reports as well. 3 Q. Do you know whether the subpostmasters were trained to 4 use a reporting facility within Horizon to generate 5 material of that kind? 6 A. I don't. I have seen the -- there was a pack of 7 training material that detailed usage of some of these 8 reports. 9 Q. Was the three species of information that you set out 10 there, information that you or your colleagues at 11 Fujitsu could generate with relative ease? 12 A. Yes, they would be in the message store. 13 Q. Were you sometimes asked to provide that information? 14 A. We probably provided it in incidents where we were 15 investigating. I can't give you any examples but I'm 16 sure we would have pulled that information out. 17 Q. Can we turn, still in connection with the species of 18 data available to Fujitsu and that which was passed to 19 the Post Office, to some emails concerning ARQ 20 filtering. Can we start, please, by looking at 21 FUJ00230912. 22 This is a series of emails between you, Steve Parker 23 and Anne Chambers on 14 May 2010, which seems to 24 reference how filters are applied on ARQ requests 25 concerning events? 52 1 A. Yes. 2 Q. Can we start on page 3, please. At the foot of the page 3 the originating email from Mr Parker to you and Anne 4 Chambers with the heading "ARQ and event filtering". He 5 says: 6 "The event lists we are being asked to check on 7 [that's Horizon Online] ARQ requests are just 8 unmanageable (7-10,000 rows in the SYSMAN3 details)." 9 Can you explain what SYSMAN3 was, please? 10 A. It was the version of the Tivoli system which was the 11 one that harvested the events from counters and the data 12 centre. 13 Q. So what's the issue that Mr Parker is raising there? 14 A. We used to get Excel spreadsheets passed to the SSC with 15 events that had been harvested in a date range and asked 16 would these events be of any -- have any impact upon 17 a counter? And, because it was from the data centre as 18 well as the counter, it was a lot of events could have 19 happened during that period. 20 Q. Why were you being asked to check for events? 21 A. I'm not totally sure but they were using the SSC as 22 people who may be able to say whether an event may have 23 been an important one impacting a counter. That was my 24 understanding, and -- 25 Q. The "they" in that sentence, who was doing the asking? 53 1 A. The -- it was the Security Team, the people who would 2 handle the ARQs. 3 Q. Security Team within Fujitsu? 4 A. Yes. 5 Q. So they were asking you to look at a lot of data -- 6 A. Yes. 7 Q. -- and see whether there was anything in the data which 8 might contain a relevant event, an occurrence, that 9 impacted on the, what, integrity or reliability of the 10 data? 11 A. The operation, I would say, yes. So we would get 12 a large Excel spreadsheet here, saying 7,000 to 10,000 13 events on it and asked to filter those to see if any 14 could have an impact on the counter's operation. It was 15 a lot of data, it took a lot of time. We generally used 16 the KEL system to say, "Go to event 1, is that in the 17 KEL?" If not, that takes out, say -- you would order 18 them to 1,000 and then "Go to another event, is that 19 going to be of any -- problematic?" No. That might 20 take out 500. 21 Then you'd keep going until you've got, say, a page 22 of events and then try to work out if those may have had 23 any impact on the counter. 24 Q. So you were being asked to, essentially, vouchsafe the 25 data that was going to ultimately be provided to the 54 1 Post Office to see whether it included any events that 2 would affect the reliability of the data? 3 A. I believe so. 4 Q. Did you know what the data was being used for, the end 5 use of it, ie in investigations and prosecutions? 6 A. I didn't know about prosecutions but we did know that 7 this was going back to Post Office. 8 Q. What did you think it was going back to Post Office for? 9 A. For when someone has requested was this counter working 10 correctly? 11 Q. Why would they want to know whether a counter was 12 working correctly? 13 A. That's my day job. 14 Q. Sorry? 15 A. It's part of my day job. 16 Q. Yes, but why did the Post Office want to know whether 17 a counter was working correctly? 18 A. I don't know. You'd have to ask them. 19 Q. Why did you think they wanted to know whether a counter 20 was working correctly? 21 A. I imagine that they had a query, saying was this counter 22 working correctly at this time and, therefore, they have 23 got a specific request about a counter not functioning 24 correctly at that time. 25 Q. The purpose of asking, Mr Simpkins, is whether you knew 55 1 that the exercise you were engaged in may result in 2 an answer or an assertion that was being fed to the Post 3 Office and they would use the product of the work that 4 Fujitsu had done, including your work, to base 5 a criminal investigation or criminal prosecution. Did 6 you know that -- 7 I -- 8 Q. -- by this time, May 2010? 9 A. I'm not sure if I knew it would go back for a criminal 10 or civil investigation but I knew that it was going back 11 to the Post Office. 12 Q. The email continues: 13 "We are allowed to filter out where the event is 14 known to have no financial impact on the counter." 15 What does that mean? Who was doing the allowing 16 there? Who said it was okay to filter out events that 17 were said to have no financial impact on the counter? 18 A. I believe that we were being asked by the Security Team 19 to do this filtering. 20 Q. Mr Parker there says "We are allowed to filter out 21 things that are known to have no financial impact". 22 A. Yes. 23 Q. Do you know who granted that permission? 24 A. No. I know where the request was coming -- from the 25 Security Team on the PinICL. 56 1 Q. It says that permission has been granted where the event 2 is known to have no financial impact on the counter. Do 3 you know why you were allowed to filter out such known 4 events? 5 A. I think we were trying to help by reducing the quantity 6 of events that will be sent back to the Post Office. So 7 there's 10,000 events here. If we can help say "These 8 ones are known to be benign from our systems", then 9 only, say, 500 or something events might go back. 10 Q. Is it right that you don't understand how that agreement 11 or position had been reached, ie this level of filtering 12 out was permissible? 13 A. No, I don't know how that got reached. 14 Q. You know that the practical effect was to reduce a big 15 number of events down to a smaller number of events? 16 A. Correct. 17 Q. The email continues: 18 "We need to get the ARQ filters up to date for 19 [Horizon Online] quickly to make the situation 20 manageable." 21 What does that mean? 22 A. So we could feed back to Gerald and his team the events 23 that we believe are benign and they would hard code 24 a change in their filters to take those events out. 25 Q. Is it right that that implies there was a lag in 57 1 recognising, for the purposes of Horizon Online, event 2 filters? 3 A. Yes. So I believe that, because it was a totally new 4 from the ground up system, there was suddenly a lot of 5 events written by a lot of new data centre servers, and 6 no thought had been done to which of these events could 7 be filtered. 8 Q. Was a record kept by you of the steps taken in this 9 filtering process? 10 A. There probably was a work instruction or a how to help. 11 Q. That's a slightly different issue. That's was there 12 an instruction on how to do it. 13 A. Yes. 14 Q. I'm asking, in each individual case, did you retain, did 15 you keep, a record of "This is the data that I started 16 with, these were the filters I applied, these are the 17 products that I ended up with that will get passed to 18 the Post Office"? 19 A. I would have to look at the PEAKs to see what we did but 20 I believe that we did feed back to Gerald "These are 21 events that we believe are benign to add to the filter", 22 but I don't know what we would have recorded on the PEAK 23 as to which events we have selected out of those 10,000 24 to filter. 25 Q. The email continues: 58 1 "According to Gerald Barnes, the way to get the 2 filters changed is: 3 "'The events need all to be checked by someone who 4 understands them. Whilst doing this they may well 5 identify certain patterns which they know to be benign. 6 They should then raise a PEAK stating which patterns 7 they consider benign and assign it to the Audit Team. 8 I will then alter our filters to ensure that these 9 events are always filtered out. This seems a little 10 tedious but it has the advantage that we have an audit 11 trail for the reason behind filtering out particular 12 events'. 13 "Can you cooperate on looking at these event lists 14 and getting the PEAKs raised into audit. Suggest John 15 ..." 16 I think that's you referred to. 17 A. Mm. 18 Q. "... runs the list and Anne add viruses on counter 19 events. If you supply me with the PEAK numbers I'll get 20 them put through. This is likely to be an iterative 21 process until we can get the events driven down. 22 "[Sample ARQs attached]. There are some obvious 23 ones on the list that can be knocked off quickly." 24 Then if we scroll up the page to page 2, please. 25 I think we see your reply. Your comments: 59 1 "The full event text was not included in the sample, 2 most events are probably not ['worth', I think that's 3 meant to say] keeping unless they specify a specific 4 transaction/journal number such that it can be tied back 5 to a financial issue. 6 "I suggest removing the following events ..." 7 Then there's a big long list. 8 You're saying apply these filters, essentially, to 9 remove events from the ARQ data? 10 A. Correct. 11 Q. Top of the page. Mr Parker replies: 12 "If you agree, let's get the necessary PEAKs raised 13 ... 14 "I'm concerned that some of the events are not 15 complete (full event text) so unable to classify." 16 What does that mean? 17 A. I think I mentioned in the part below that the full 18 event text was not supplied to us. So the event text 19 was truncated in some way when the -- it was extracted 20 to us. 21 Q. Up the page, please. We can see Ms Chambers's reply: 22 "Counter events -- I think we should apply the same 23 filters to SYSMAN3 as have already been applied to 24 SYSMAN2 ... However I don't have a list of these. I'm 25 reluctant to put much effort into justifying in this 60 1 area." 2 What was she meaning by that, please? 3 A. SYSMAN2 was the previous version so I'm presuming that 4 she's saying that there was already a filter set up for 5 the events from the SYSMAN2 product and can that just be 6 brought forwarded to the SYSMAN3 product. 7 Q. So, overall, this is a discussion within the SSC as to 8 the filters that are going to be applied to ARQ data -- 9 A. Events, yes. 10 Q. -- yes -- to events within ARQ data to reduce the 11 volume? 12 A. Correct. 13 Q. How was it established if an event had no known 14 financial impact? 15 A. I think, when I did it, I normally started with the KELs 16 and searched through the events that were in the Excel 17 spreadsheet against the KEL database. 18 Q. Isn't that a bit of a shaky way of doing things? 19 Doesn't it rely on the accuracy and completeness of the 20 KELs? 21 A. It did give you a good starting point. You could search 22 the PEAKs. There should, in theory, almost have been 23 a KEL for every single event raised. So the KELs, as 24 I said, is a misnomer it's a knowledge article base, and 25 the SMC, who were the second line support team, whenever 61 1 they met an event that was not already KEL'd for and not 2 already filtered, they would raise a call for it so 3 there were a lot of knowledge articles all about the 4 events in the data centre. 5 Q. That can come down. Thank you. 6 So, essentially, technical decisions were being 7 taken on what could or could not evidence a problem with 8 financial information. Was input provided by the Post 9 Office on this, to your knowledge? 10 A. Not to my knowledge, no. 11 Q. Was this an exercise conducted, therefore, only 12 internally by Fujitsu? 13 A. Yes, I believe so. 14 Q. To your knowledge, was the Post Office told the outcome 15 of the exercise, ie what filters had been applied to 16 filter out material that wouldn't be checked? 17 A. I don't know. 18 Q. To your knowledge, was there such a discussion? 19 A. I don't know. 20 Q. The decision was ultimately taken to use the previous 21 iteration of SYSMAN: SYSMAN2? 22 A. It was mentioned in the email. Whether it was carried 23 forward, I can't tell you. 24 Q. Well, do you know -- 25 A. I don't know. 62 1 Q. The email tended to suggest that we should just use 2 SYSMAN2 and that Anne Chambers was reluctant to put much 3 effort into justifying each additional exclusion. 4 A. That's what the email said would probably ask Gerald. 5 He would be able to tell you what filters were applied. 6 Q. Mr Parker had suggested that conducting some new checks 7 would be helpful, hadn't he? 8 A. Yes. 9 Q. To your knowledge was that done? 10 A. We did add new checks -- sorry, we added -- I mean we 11 added information back to Gerald about the new events, 12 yes. 13 Q. Can we turn, please, to FUJ00228917. This is an email 14 exchange of a year later. You'll see that it again 15 involves you. It's a one-page email exchange. It's 16 quite difficult to ascertain what's happening, certainly 17 to an outsider, but, if we look at the bottom of the 18 page, please, an email from John Rogers, who is 19 described as the test lead for Horizon Online. What's 20 "LST"? 21 A. Live service test or system test. 22 Q. The subject line is "ARQ retrieval format inadequate for 23 support use": 24 "Steve 25 "This new functionality is under test ... 63 1 "Have you seen the new spreadsheet that is produced? 2 "... are you happy with the format? 3 "If not would you like to see an example?" 4 Up the page, please: 5 "Have you got an example please ..." 6 He copies you in: 7 "[he has] not seen it at all! 8 Then at the top of the page, please: 9 "Attached is a copy of the output events file for 10 two ARQs. 11 "[One] contains SYSMAN2 data ... 12 "[The other] contains SYSMAN3 data." 13 Is this email chain -- and this is all we've got -- 14 evidence of some exploration of how ARQ retrieval could 15 be used by the SSC? 16 A. I would say this is them making a change to the live 17 system and it's currently in live -- live support 18 test -- live service test, system test -- before it goes 19 live, and they are checking that we are happy with the 20 output of that change to the event spreadsheet. 21 Q. You told us earlier this morning that you in the SSC did 22 not use ARQ data for the purposes of your work? 23 A. This is not for our work. 24 Q. This is not for your work -- 25 A. No. 64 1 Q. -- but the area of your work that we're talking about 2 now is SSC's involvement in the filtering of ARQ data? 3 A. Yes. 4 Q. This is talking about, is this right, under Horizon 5 Online, the filtering of output events -- 6 A. Yes. 7 Q. -- and the SSC being given an opportunity to inform or 8 configure the format of the retrieval to increase its 9 usefulness? 10 A. I presume so, yes. 11 Q. Would that be usefulness not to the SSC but usefulness 12 to the end user, the Post Office? 13 A. I imagine it could be the SSC, to help us do the 14 filtering that we are previously doing. I don't know 15 what the instigation of the change was. 16 Q. That was my next question: what was the outcome of this? 17 A. I don't know. 18 Q. Are you aware whether this discussion was ever 19 communicated back to the Post Office, ie "Under Horizon 20 Online there's an opportunity to change the filtering 21 process of the ARQ data, we're going to now apply these 22 filters going forwards from January 2011"? 23 A. I don't know of that. That may be a good one to ask 24 Gerald. Have we looked at the PEAK? It's got a PEAK 25 reference related to this, 206531. 65 1 Q. If I have it's presently lost in my memory somewhere but 2 we can do that and maybe do that with Mr Barnes? 3 A. Yeah, it may specify why this change is coming about and 4 what the outcome was. 5 Q. Again, at this stage, did you know what the Post Office 6 was using the ARQ data for? 7 A. I don't believe so. I'm not sure when we would have 8 been aware of -- there were prosecutions going on. We 9 did -- as I was saying, we did stop doing this and that 10 must have been when we were aware. So the SSC decided 11 we're not happy doing this filtration if it's going to 12 be used in court cases, and we stopped. 13 Q. Why weren't you happy? 14 A. Because it -- again, leading on from Anne having to give 15 evidence, we thought that if it -- we were making the 16 filtered choices, they may want someone to come up and 17 explain exactly why in a court case. 18 Q. Why would you be unhappy about doing that? 19 A. I think it was just we did not wish to do that. 20 Q. But why? 21 A. Because it's -- I would say that it would be difficult 22 to explain technically every single decision you've made 23 out of 10,000 events, why you decided to filter that. 24 Q. That document can come down. Thank you. 25 Why would it be difficult to explain to a court in 66 1 a statement or in oral evidence why you had made 2 filtering decisions? 3 A. I guess that you would have to refer to documentation, 4 to examples of PEAKs, to examples of KELs for every 5 single one, and we felt that it's something that just 6 gives them everything. 7 Q. Gives the postmasters everything? 8 A. Well, in the ARQ, have all of the events and then, if 9 you wish to ask questions about individual events, we 10 can do that, rather than us filtering them. 11 Q. What's the problem with giving them everything? 12 A. There is no problem giving them anything. We were 13 helping to do the filtering, now we've made a decision 14 not to do the filtering and then you can ask about 15 individual events instead. 16 Q. So have I understood you correctly to say that there 17 came a realisation within the SSC, a point in time at 18 which you realised the use to which your product was 19 being put? 20 A. Which to our filtering has been put is a good way of 21 putting it, and we decided that that's not what we want 22 to do. 23 Q. When was that? 24 A. I cannot tell you, I'm afraid. I could probably try and 25 find out by talking to the Security Team. 67 1 Q. It must have been after January 2011? 2 A. Indeed, otherwise that email wouldn't have been sent. 3 Q. Who made the decision? 4 A. I think it was the SSC Team Leaders and the SSC Team 5 Manager. 6 Q. So Mr Parker? 7 A. I think that the SSC Team Leaders pushed with Mr Parker 8 agreeing. 9 Q. Why did the Team Leaders have to push Mr Parker to 10 agree? 11 A. I think it's just we were doing this process and then 12 suddenly there's this realisation to say "Can we not do 13 this process?" 14 Q. Does it follow that, before the SSC sort of downed tools 15 on this aspect of its work, none of you had been asked 16 to explain in any formal way, to either the Security 17 Team or to the Post Office, what you were doing and what 18 filtering had occurred? 19 A. I imagine you're correct, yes. I cannot recall having 20 done that and I don't know the latter bit about which 21 filtering has occurred. That is probably because you 22 had the filtering in already as well. So I don't know 23 about that part. 24 Q. Would you understand that, if a court is presented with 25 a set of data, it would want to know what has been done 68 1 and each of the steps that have been taken to produce 2 that set of data? 3 A. Totally. 4 Q. That was, is this right, what led the SSC to down tools, 5 as I've described it? 6 A. Yes, I think that's fair. 7 Q. Was that just a reluctance to be dragged into or become 8 involved in court proceedings or was it because of 9 difficulties in explaining the nature of the exercise 10 that you were undertaking? 11 A. I imagine it was a -- I would say it's a partial both 12 but I would say that it made a lot more sense to give 13 them the full events than to give them a filtered 14 version. 15 Q. Can I just try and understand what you've just said 16 there. What you've told us in your second witness 17 statement, that the filtered ARQ information that was 18 provided to subpostmasters does not contain sufficient 19 information for the postmaster to assess the health of 20 the Horizon system as it affected their branch, correct? 21 A. Correct, but when I made that statement I was looking at 22 events and transactions. I was not thinking about 23 events -- sorry, counter events, not Windows and 24 operating system events, which is what we're talking 25 about now. 69 1 There was -- in that witness statement, there were 2 two pieces of evidence shown to me: one were counter 3 events, ie logon, logoffs, things like that; and the 4 other one was transactions. 5 Q. Just to try to sum up this part of your evidence: was it 6 "We don't want any involvement with the SSC in court 7 proceedings after what happened to Anne"; was it, "We're 8 unhappy about the exercise we're being asked to 9 undertake and we wouldn't want that being explored in 10 court"; or "We know that there's more information that 11 could be revealed to subpostmasters to show the health 12 of the system"? 13 A. I think it was partially the first, Anne, and then also 14 partially that it is a manual process and that you can 15 obviously make mistakes. 16 Q. So does that mean you wouldn't want your homework 17 subject to scrutiny in a court? 18 A. No, I'm happy to have my homework scrutiny'd in a law of 19 court (sic), and I could go through and explain the 20 reason why for each of them but would you be hauled over 21 the coals if you had made a mistake or if an event that 22 was, according to a KEL, not financially impacting, 23 later on becomes financially affecting because there's 24 been a change? 25 Q. You and your colleagues must have been sufficiently 70 1 concerned that that was a realistic possibility to 2 include that in your reasoning for not wishing to do it? 3 A. Correct. 4 Q. Can I turn lastly on this topic, please, to FUJ00225729. 5 This series of emails, again involving you, this is 6 October 2010, concerns the investigation of an issue of 7 system integrity at the Ferndown sub post office. 8 Ms Penny Thomas asked for an investigation to be 9 undertaken and you become involved in it. 10 Can we start with page 3 of the email chain, please, 11 which is the originating email, an email from Emma 12 Langfield, the Line Service Team within Post Office, to 13 Mr Thompson, copied to Ms Thomas and David Hulbert: 14 "... I hope today's meeting ... proved to be 15 beneficial. 16 "My apologies for the late notification of the 17 following but I am hoping that you will be able to 18 assist in a rapid turnaround for an ARQ request. 19 "Our Security Team, who forward ARQ requests to 20 yourselves for extraction ... have this afternoon sent 21 an emergency ARQ to Penny's team for processing. This 22 has come from Lynn Hobbs, Branch Network Manager, which 23 in turn was passed into Lynn by Paula Vennells, Post 24 Office Limited Network Director. 25 "This request is a data extract for the above branch 71 1 from 1 September 2009 to 30 September 2010. 2 I understand from Mark Dinsdale that the agreed 3 turnaround for ARQ requests is 7-14 working days, but 4 the ARQ above ... is a business priority. 5 "Given the resource at your disposal and your team's 6 ... workload is there any way that the 12-month extract 7 can be completed [by] Monday, 4 October ..." 8 I think this is being sent at 5.48 on the Friday: 9 "We have Helen Rose [Post Office Limited] on standby 10 to decipher the data and this will be her priority when 11 received, but we need to feed back a delivery date and 12 time to Mark, Lynn and Paula." 13 Firstly, had you any awareness of Helen Rose within 14 the Post Office? 15 A. No. 16 Q. Did you have any dealings with her or any understanding 17 of her competency to decipher ARQ data? 18 A. No. 19 Q. Did any members of the SSC, to your knowledge, give 20 training or assistance to anyone within the Post Office 21 on deciphering ARQ data? 22 A. Not to my knowledge. 23 Q. Thank you. If we can scroll up, please, Penny Thomas to 24 Peter Thompson, copied to Donna Munro: 25 "We are looking at a request for 13 months of data 72 1 received at 4.30 on Friday afternoon. It is not 2 possible to return this request today. 3 "I will provide an update with an estimated return 4 time frame later in the day." 5 Further up the page, please: 6 "Can you inform the customer of the perceived time 7 scales at this initial stage just to provide some 8 perspective of time scales." 9 Penny Thomas: 10 "I can't do more until we identify the size of 11 outlet and number to counters." 12 Up the page, please, it gets sent on to Steve 13 Parker: 14 "Please see [the] string. A forewarning that we 15 will be sending SSC thirteen months worth of events for 16 this outlet ... Would SSC please be able to review and 17 return comments asap?" 18 Further up the page: 19 "... Steve is [out of office] -- I'm not sure who to 20 forward this to -- but this really urgent ..." 21 You, Anne Chambers and some others get copied in, 22 yes? 23 A. Yes. 24 Q. Further up the page, you reply: 25 "Of course we can look at the provided data but it 73 1 will take some time to trawl through the potential 2 number of events. 3 "The comment in the trail below 'We have Helen Rose 4 on standby to decipher the data and this will be her 5 priority ...' implies that they would like to do the 6 trawl themselves." 7 What did you mean by that? 8 A. It means that they wanted to go through the events 9 themselves. 10 Q. Ie they didn't want analysis by -- 11 A. They didn't want the filtered -- 12 Q. The filters by you? 13 A. Correct. 14 Q. Then further up the page, please, Ms Thomas says: 15 "We, as a matter of course, check all system events 16 before returning transaction records to [the Post 17 Office]." 18 Is that the exercise that you have just described to 19 us? 20 A. Yes. 21 Q. "Their trawl is to do with transaction records ..." 22 So she's essentially saying "No, you still need to 23 do the filtering first". 24 A. Correct. 25 Q. Is that right? 74 1 A. Yes. 2 Q. "Their trawl is to do with transaction records, which, 3 I'm sure you're aware is a totally different kettle of 4 fish." 5 Then further up the page the PEAK numbers are set 6 out and: 7 "... there is a lot of senior management focus on 8 this request from both Fujitsu and [Post Office] so 9 [please treat it] as a priority." 10 So is this just another example of the SSC being 11 asked to review ARQ data and filter it? 12 A. The events, yes. 13 Q. Yes, the events, albeit this is on the hurry-up? 14 A. Yes. 15 Q. So the request that's included in that email is 16 something that the SSC was undertaking routinely, 17 analysing events data and filtering it, in any event? 18 A. Correct. 19 Q. So we've got to the situation, is this right, 20 Mr Simpkins, where you say, in order to look at the 21 health of the system from a postmaster's perspective, 22 you would not use the filtered ARQ data to do so? 23 A. If I was taking a call from a postmaster, correct. 24 Q. But then what's provided to the Post Office is the 25 filtered ARQ data? 75 1 A. Correct. 2 Q. Were those two worlds ever compared with each other: 3 I wouldn't look at that data if I wanted properly to 4 investigate the health of the system; I'm going to 5 provide that data to the Post Office? 6 A. I don't know about how the ARQ process was designed or 7 created or agreed with the Post Office. The filtering 8 of the events is effectively something I would be doing. 9 If a call came in for a subpostmaster, I would go 10 through the events that's happened in the data centre to 11 see if I could see anything that may have affected them. 12 So that filtering of events is kind of something 13 I would do, if I had a call from a subpostmaster. The 14 ARQ -- the selection of attributes to return to the 15 subpostmaster, I don't know how that got agreed. 16 Q. But you just did it because it was part of your 17 function? 18 A. No, sorry, the events bit, we're talking about the 19 events part, that is something I would do anyway if 20 I had a normal call. So that's totally -- I'm totally 21 happy with that. The other parts, the transactional 22 part and the message store filtering, we didn't do, and 23 I don't know how that got agreed between Fujitsu and the 24 Post Office about how -- what form they wanted that ARQ 25 in. 76 1 Q. Got it, understood. 2 Can we turn to a new topic, please, which is remote 3 access. Can we start by looking at FUJ00088036. Now, 4 I asked you some detailed questions about this on the 5 last occasion, so I'm not going to go over all of what 6 you said but I just want to refresh in our minds what 7 you said about it, please. So this is an outline of the 8 secure support system of 2 August 2002; can you see 9 that? 10 A. Yes. 11 Q. If we look at further down on page 1, we can see that 12 one of the approval authorities is Mr Peach, the SSC 13 Manager, right at the foot of the page, yes? 14 A. Yes. 15 Q. Then, over the page to the second page, in the second 16 box down, we can see that reviewers included Mr Peach -- 17 just scroll down a little bit please, thank you -- we 18 can see that reviewers included Mr Peach and Mr Parker; 19 can you see that? 20 A. Yes. 21 Q. If we can go, please, to page 13, the document describes 22 some "Areas of Concern" at 4.1: 23 "There are two major areas of concern with the 24 current support processes: 25 "1. Second line support does not have the tools 77 1 necessary to perform their function ..." 2 Then this: 3 "2. Third line and operational support 4 organisations' access to the live system is not fully 5 audited and in some cases is unrestricted in the actions 6 that can be carried out ..." 7 That's describing that second point there, the 8 position in the SSC; is that right? 9 A. Yes, at that time, yes. 10 Q. Then at 4.1.2, if we just scroll down a little bit: 11 "Third line support staff receive repeat instances 12 of calls that should have been filtered out by second 13 line ... 14 "The current ... practices were developed on a needs 15 must basis; third line support diagnosticians had no 16 alternative other than to adopt the approach taken given 17 the needs to support the deployed Horizon solution. 18 "The consequences of limited audit and system ... 19 access afforded to third line support staff provide the 20 opportunity to: 21 "Commit fraudulent acts; 22 "Maliciously or inadvertently affect the stability 23 of the new Network banking and Debit Card online 24 services; 25 "In addition a complete audit would allow Pathway to 78 1 defend SSC against accusations of fraud or misuse." 2 Then on to page 15, please. 3 A. I did also comment on this last time and say I don't 4 agree with the "commit fraud" on that, when I was last 5 here. 6 Q. Yes. 7 A. Okay. 8 Q. 4.3.2, at the top of the page, please, describing third 9 line support: 10 "All support access to the Horizon systems is from 11 physically secure areas. Individuals ... in the support 12 process undergo more frequent accurate vetting checks. 13 Other than the above[,] controls are vested in manual 14 procedures, requiring managerial sign-off controlling 15 access to post office counters where update of data is 16 required. Otherwise third line support has: 17 "Unrestricted and unaudited privileged access 18 (system admin) to all systems including post office 19 counter PCs; 20 "The ability to distribute diagnostic information 21 outside of the secure environment; this information can 22 include personal data ... business sensitive data and 23 cryptographic ... information. 24 "The current support practices were developed on 25 a needs must basis; third line support diagnosticians 79 1 had no alternative other than to adopt the approach 2 taken given the need to support the deployed Horizon 3 solution. 4 "There are no automatic controls in place to audit 5 and restrict user access. This exposes Fujitsu ... to 6 the following potential risks: 7 "Opportunity for financial fraud; 8 "Operational risk -- errors as a result of the 9 manual actions causing loss of service to outlets; 10 "Infringements of the Data Protection Act." 11 Is what it described there in paragraphs 4.3.2 12 accurate as representing the position in August 2002? 13 A. I don't agree with the opportunity for financial fraud. 14 Otherwise -- oh, and the cryptographic key information, 15 we didn't have access to. 16 Q. Is, overall, what is being described here the facility 17 for third line support to have remote access to the 18 Horizon system? 19 A. We had remote access to the live system. 20 Q. It includes that this is unrestricted and unaudited 21 access; is that accurate? 22 A. There were definitely events written whenever we 23 connected. So at this time, we used some software 24 called Rclient to connect. 25 Q. Capital R, capital C (sic), client? 80 1 A. Yeah. Sorry, yes, Rclient, and it would have written 2 a Windows event when we had written to the counters or 3 data centres. It would have also -- before we got 4 there, as I said, we connected to the data centre and 5 that would have also been audited as well. So it's not 6 unaudited but I don't believe it would show you who 7 connected, which person. 8 Q. When you say it's not unaudited, it's not unauditable; 9 is that right? 10 A. It says "unaudited privileged access". 11 Q. Yes, but what you've just described is that the 12 situation was that it could have been audited? 13 A. Yes. 14 Q. Was it, in fact, audited? 15 A. I don't know. We -- 16 Q. Was it unrestricted? 17 A. Yes, I believe we had admin access, which is effectively 18 the highest level. 19 Q. Do you know why Mr Peach would have authorised the 20 issuing of this document; Mr Peach and Mr Parker would 21 have reviewed the document and let things remain in it 22 that, in your view, are not accurate? 23 A. I can't comment about that, I'm afraid. I can just give 24 you mine. 25 Q. Sorry? 81 1 A. I cannot comment about their review process. I can just 2 give you mine. 3 Q. For how long did what's described in this document 4 remain the position after August 2002? 5 A. I did a little research when I saw this in my pack and 6 I found a PEAK that said in -- was defining in July 2003 7 the new SSH -- OpenSSH was being used and there was 8 a peak on an issue the SSC had with it so we were 9 definitely using it by July 2003. 10 Q. What was the new system? 11 A. So the new system used something called OpenSSH and it 12 allowed us to log every single key press that the third 13 line support person made when connecting down to the 14 counter. 15 Q. When you say it allowed you to -- 16 A. Sorry, it was. 17 Q. It was. 18 A. The software was recording every single key press to 19 an auditable file. 20 Q. So that was in place from at least July 2003? 21 A. Yeah, I'm sure you could probably find out the -- once 22 you know what release package that went under, you 23 should be able to find out the exact date. But, as 24 I say, I found that PEAK and so I know it was working 25 from July 2003. 82 1 Q. Was it recognised within the SSC at this time that the 2 privileged access that the 25-odd members of the SSC had 3 was an uncomfortable position to be in? 4 A. Probably when it was pointed out because support 5 wouldn't know what operating system logging and 6 everything else around us was in place. We were told, 7 "How can you connect to the counter? This is how you 8 connect to the counter, this is how you do your job". 9 When it was pointed out, I imagine, yes, they would be 10 uncomfortable with it. The new version gave us better 11 wrapper around our commands, so we actually had more 12 facilities with the new OpenSSH, we had a Cygwin shell 13 down there, which we connected to, and it was nice 14 enough for support to use, overall. 15 Q. Can we move forwarded eight years or so, until October 16 2010, and look at POL00117863, please. 17 This is a document that isn't dated but, from other 18 evidence, looks to have been created for the purposes of 19 a meeting held at the beginning of October 2010. We can 20 see that there are four Fujitsu employees attending or 21 proposed to attend the meeting, and they included you; 22 can you see that? 23 A. Yes, I presume this was created by Post Office as they 24 got my role incorrect. 25 Q. Because it records you as being a member of Security? 83 1 A. Correct. 2 Q. Unfortunately, the document is not authored and, as I've 3 said, not dated. Did you, in fact, attend this meeting? 4 A. I can't remember. 5 Q. Let's look at what is recorded, and this seems to be 6 a note prepared for the purposes of a meeting, rather 7 than a record of the meeting. 8 A. Okay. 9 Q. "What's the issue?" 10 The Chairman is very familiar with this document. 11 I want to use it for a purpose different than that which 12 it is usually used for. 13 "What's the issue? 14 "Discrepancies showing at the Horizon counter 15 disappear when the branch follows certain process steps, 16 but still show within the back end branch account ... 17 currently impacting [around] 40 branches since migration 18 onto Horizon Online, with an overall cash value of 19 [around] £20,000 loss. This issue will only occur if 20 a branch cancels the completion of the trading period, 21 but within the same session continues to roll into a new 22 balance period." 23 So, overall, would you agree that that describes 24 what is known as the receipts and payments mismatch bug? 25 A. Yes, I commented on this last time I was here, as well. 84 1 One of my -- 2 Q. I don't think you commented on this document last time? 3 A. No, I did comment on the receipts and payments, I think 4 it was when the Core representatives asked me questions. 5 Q. Yes. 6 A. One clarification I made at that time was that this is 7 visible to the subpostmaster. So it's visible on the 8 balance report they print out, because there's 9 a difference between receipts and payments, and it's 10 also visible when they roll the branch trading statement 11 because they will get a non-zero trading position, and 12 that seems to have not been picked up from my last -- 13 when I was last here, because it's been referred to as 14 they can't see this. 15 I've got some PEAKs where I can demonstrate the 16 subpostmaster got that non-zero trading position, rang 17 in, and we're told it's related to this, and that it's 18 with Post Office and they know about it. 19 Q. If we go to page 2, please, and look a third of the way 20 down, the part that I think is emboldened: 21 "Note the branch will not get a note from the system 22 to say that there is Receipts and Payment mismatch, 23 therefore they will believe they have balanced 24 correctly." 25 Is that accurate. 85 1 A. Yes, at a point in you time. So what happens is you are 2 rolling your stock unit, you go to roll your stock unit 3 and you get a discrepancy warning. You cancel, you go 4 back to the previous screen. Then you carry on anyway 5 and it's lost that discrepancy and you have a receipts 6 and payments mismatch, but because you're passed the 7 trial balance, it doesn't tell you. That's this point. 8 It does print it out. Later you will roll your 9 branch, which is all the stock units added together. 10 That takes all the stock units and realises that they 11 don't add up. That's when you get the non-zero trading 12 position error reported to the subpostmaster. 13 Q. So they'll see there's an error but they won't know the 14 cause of it? 15 A. They will see the receipts -- it's basically telling you 16 the receipts and payments for all your stock units do 17 not add up to zero. 18 Q. Moving down to the "Impact": 19 "The branch has appeared to have balanced, whereas 20 in fact they could have a loss or a gain ..." 21 Accurate? 22 A. Correct. 23 Q. "Our accounting systems will be out of sync with what's 24 recorded at the branch ..." 25 Correct? 86 1 A. That's Post Office's side. I believe that's correct. 2 I couldn't tell you. 3 Q. "If widely known could cause a loss of confidence in the 4 Horizon system by branches. 5 "Potential impact on ongoing legal cases where the 6 branches are disputing the integrity of Horizon data. 7 "It could provide branches ammunition to blame 8 Horizon for future discrepancies." 9 Were these concerns of yours, these last three? 10 A. No. 11 Q. They were concerns of other people at the meeting, were 12 they, presumably? 13 A. Presumably whoever called the meeting, yes. 14 Q. Over the page, please, top of the page: 15 "The Receipts and Payment mismatch will result in 16 an error code being generated which will allow Fujitsu 17 to isolate branches affected by this problem, although 18 this is not seen by the branches." 19 Accurate? 20 A. As I say, again, it's twice it tells them. Once on the 21 stock unit balance report, it's got the mismatch on that 22 report, and when they roll the branch it will tell them. 23 Q. So that's inaccurate? 24 A. Yes. 25 Q. "We [that tends to suggest this was written by the Post 87 1 Office] have asked Fujitsu why it has taken so long to 2 react to and escalate an issue which began in May", 3 they're going to get back to the Post Office. 4 "Fujitsu are writing a code fix which will stop the 5 discrepancy disappearing from Horizon in the future. 6 They are aiming to deliver this into test week [of] 7 4 October ... 8 "The code fix will stop the issue occurring in the 9 future but it will not fix any current mismatch at 10 branch." 11 Accurate? 12 A. Yes. 13 Q. "Proposal for affected Branches", if we go down, please, 14 and look at solutions 1, 2 and 3: 15 "There are three potential solutions to apply to the 16 impacted branches." 17 The recommendation is that 2 should be adopted: 18 "SOLUTION ONE -- Alter the Horizon branch figure at 19 the counter to show the discrepancy. Fujitsu would have 20 to manually write an entry value to the local branch 21 account." 22 Under "Risk": 23 "This has significant data integrity concerns and 24 could lead to questions of 'tampering' with the branch 25 system and could generate questions around how the 88 1 discrepancy was caused. This solution could have moral 2 implications of Post Office changing branch data without 3 informing the branch." 4 So does that reflect the fact that, at this time, in 5 October 2010, Fujitsu had the ability to manually write 6 entries into local branch accounts and that would not be 7 visible to the subpostmaster? 8 A. So this is HNG-X, so this is the branch database. So 9 Fujitsu could make up entries to the branch database. 10 Q. Without the subpostmaster knowing about it? 11 A. Yes. 12 Q. So, in that sense, it would be covert, wouldn't it? 13 A. If you don't tell someone about it, I guess that is -- 14 Q. Yes, it would be completely invisible to the 15 subpostmaster that Fujitsu had been inserting values 16 into their accounts? 17 A. Yes, it could be invisible, if they -- it does say that 18 the -- if they've already rolled, then there's going to 19 be -- they would have already known that there was 20 issues but, yes, you could not see potentially us 21 inserting an update into a database, that is totally 22 separate from the counter. 23 Q. So, as at October 2010, Fujitsu retained the facility of 24 remote access to write entry values to local branch 25 accounts covertly without a subpostmaster knowing? 89 1 A. It's not remote access. We're in the data centre. So 2 it's the branch database where this change will take 3 place in HNG-X. 4 Q. But, nonetheless, the facility to write entries into 5 accounts which have the effect of changing financial 6 information covertly, ie without the subpostmaster 7 knowing it has even occurred? 8 A. Yes. 9 Q. So I'm not looking about whether it was done and whether 10 it was revealed or not, I'm just saying this is a record 11 for a different purpose of showing that that facility 12 remained? 13 A. It is a different facility, obviously, because we're now 14 talking about a database update in the branch database 15 and we're not talking about accessing the counter at 16 all. So the counter's records are now held centrally 17 and we are talking about updating it in the branch 18 database but, yes, that is a -- there is the possibility 19 of a database update and, if you don't communicate that 20 to the subpostmaster, you're making a database update, 21 then that is correct. 22 Q. There's nothing else I want to ask about this 23 document -- we have trawled over it with other people 24 a lot -- other than which solution, 1, 2 or 3, was 25 adopted? 90 1 A. I don't believe I -- we definitely didn't do any updates 2 to the database, so I don't know which option. I think 3 "Don't do anything" was probably -- whether they updated 4 the POLSAP systems, I can't tell you. That's Post 5 Office. 6 Q. So you can't recall out of solutions 2 and 3 which was 7 adopted? 8 A. No. 9 Q. Thank you. Can we move on, please, to POL00029791. 10 This is a document that we think dates from 2014, if we 11 just go to page 10, please. We can see that the 12 facility has been used to record who made the changes 13 and the dates that they made them. Can you see that? 14 A. Yes. 15 Q. Hence why I'm suggesting that it's 2014, so in fact 16 December 2014. Back to page 1, please. It's part of 17 the review and mediation scheme, correspondence, 18 essentially, between the Post Office and Second Sight. 19 The document records that Second Sight has asked: 20 "Can Post Office or Fujitsu edit transaction data 21 without the knowledge of a subpostmaster?" 22 Then, if we go to the foot of the page, please: 23 "This document has been prepared with the assistance 24 of Fujitsu and the Post Office IT&C team. Both have 25 approved the document as being accurate." 91 1 Were you part of the group of people from Fujitsu 2 who helped to prepare the document? 3 A. No, I don't believe so. 4 Q. Have you ever seen the document before, other than in 5 preparation for this case -- 6 A. No. 7 Q. -- for this Inquiry? 8 A. No. 9 Q. Just go back to the top of the page, please: 10 "Phrasing the question in this way [that's 'Can Post 11 Office remotely access Horizon?'] does not address the 12 issue that is of concern to Second Sight and Applicants. 13 It refers generically to 'Horizon' but more particularly 14 is about the transaction data recorded by Horizon. 15 Also, the word 'access' means the ability to read 16 transaction data without editing it -- Post 17 Office/Fujitsu has always been able to access 18 transaction data however it is the alleged capacity of 19 Post Office/Fujitsu to edit transaction data that 20 appears to be of concern ... it has always been known 21 that Post Office can post additional correcting 22 transactions to a branch's accounts in ways that are 23 visible to subpostmasters (ie [TCs and TAs]) -- it is 24 the potential for any hidden method of editing data that 25 is of concern. 92 1 "In the light of these issues, Second Sight and Post 2 Office have therefore agreed the above reformulation of 3 the question to be addressed", ie can Post Office and 4 Fujitsu edit transaction data without the knowledge of 5 a subpostmaster? 6 If you had been asked that question "Can Post Office 7 or Fujitsu edit transaction data without the knowledge 8 of a subpostmaster", your answer would be yes, wouldn't 9 it? 10 A. I would say Fujitsu would be able to without the correct 11 controls. 12 Q. Fujitsu could but Post Office can't? 13 A. I can't see how Post Office can. 14 Q. Yet the answers given: 15 "In summary, Post Office confirms that neither it 16 nor Fujitsu can edit transaction data without the 17 knowledge of a subpostmaster." 18 That's just wrong, isn't it? 19 A. This is HNG-X, so, yes, it is possible with the DBA or 20 sufficient access to a database to update the database. 21 Q. So just to answer my question, that sentence, "In 22 summary, neither Post Office nor Fujitsu can edit 23 transaction data without the knowledge of 24 a subpostmaster" is wrong, isn't it? 25 A. I believe so. 93 1 Q. Over the page, please, to page 2. Just under the bullet 2 points next to edit 9, a sentence which begins, "There 3 is no functionality"; can you see that? Thank you: 4 "There is no functionality in Horizon for either 5 a branch, Post Office or Fujitsu to edit, manipulate or 6 remove a transaction once it has been recorded in 7 a branch's accounts." 8 That's wrong as well, isn't it, insofar as it 9 concerns Fujitsu? 10 A. Yeah, it's the -- the basic functionality. We did have 11 the branch -- sorry, the transaction correction tool, 12 which we used once, and I would call that functionality 13 in Horizon. The bit -- the fact that it is a database 14 and someone, a DBA could have access to it, is not 15 functionality in Horizon, if that makes sense. 16 Q. So, overall, if you had seen that sentence, you would 17 have said that is incorrect? 18 A. The functionality -- the basic functionality is that is 19 correct, you can only add using the correction tool. As 20 a DBA, you could have access to a database -- 21 Q. Thank you. 22 A. -- and update it. 23 Q. That can come down, thank you. I think it's right that 24 you didn't give evidence about remote access or any 25 evidence in the Group Litigation Order proceedings in 94 1 the High Court; is that right? 2 A. That's correct. 3 Q. But you told us on the last occasion that you and other 4 colleagues in the SSC provided information to the 5 solicitors, as you said? 6 A. That's correct. 7 Q. Was that the solicitors for the Post Office? 8 A. Yes. 9 Q. What information did you provide the solicitors to the 10 Post Office? 11 A. We were asked many questions, I believe, mainly about 12 PEAKs and about KELs, about how the system worked. 13 We -- 14 Q. Did you openly discuss the existence of KELs with the 15 solicitors for the Post Office? 16 A. I wrote a program to export them all to files so they 17 could have a copy. 18 Q. Why were you providing information to the solicitors for 19 the Post Office in the Group Litigation proceedings? 20 A. Probably two reasons: I wrote PEAK and I maintain the 21 SSC website with the KELs, the web constructions, 22 et cetera, so therefore I'm the person to export from 23 those. We are part of SSC and, therefore, a technical 24 unit with the knowledge of how the system works, and 25 Steve was giving witness evidence and -- 95 1 Q. Was there a stage when you were going to be used as 2 a witness? 3 A. I was asked if I would like to and I declined. 4 Q. You declined. Why did you decline? 5 A. I didn't want to. 6 Q. Why not? 7 A. They actually asked me if I would like to and I said no. 8 Q. Why didn't you want to give evidence? 9 A. I was not comfortable giving evidence. 10 Q. Why were you uncomfortable? 11 A. Because it's not in my skillset to give evidence. 12 Q. Or was it the substance of the evidence that you might 13 give? 14 A. No, I'm happy with lots of questions and answering 15 questions, that's my daily role. I'm more than happy to 16 do that. I don't like the environment. 17 Q. You told us on the last occasion you were aware of 18 a discussion at the time of the Group Litigation about 19 the suitability of Gareth Jenkins as a witness. Was 20 that to his suitability to give evidence as a witness in 21 the Group Litigation? 22 A. No, I don't think I commented on his suitability. 23 Q. Was that, therefore, a discussion about his past 24 suitability as a witness? 25 A. Yes, I think -- sorry, could you go through the question 96 1 again? 2 Q. Yes. You told us on the last occasion that you were 3 aware of discussion at the time of the Group Litigation 4 about the suitability of Gareth Jenkins as a witness and 5 I'm asking: is that his suitability as a witness to give 6 evidence in the Group Litigation or his past suitability 7 as a witness to give evidence in other proceedings? 8 A. No, he's definitely more than capable of giving 9 evidence. He knows his subject extremely well. That 10 was I think in reference to a document I'd seen about 11 the Post Office talking about his suitability. 12 Q. So was there a discussion in the run-up to the Group 13 Litigation trials about Mr Jenkins' suitability to give 14 evidence as a witness? 15 A. I can't recall that. He's more than -- he'd be 16 absolutely fine doing that, from what I know of him. 17 Q. What, therefore, was the discussion about, then? 18 A. Um -- 19 SIR WYN WILLIAMS: I'm sorry to interrupt but while 20 Mr Simpkins is thinking, I haven't got him on the 21 screen. 22 Now, I have. Thank you. 23 MR BEER: Thank you. What was the discussion about, then? 24 A. I cannot recall what the discussion was about because he 25 would be the perfect person to give evidence. 97 1 Q. Why did Mr Parker end up giving evidence about, amongst 2 other things, remote access and not the "perfect 3 person", Mr Jenkins? 4 A. Mr Jenkins is the architect. Mr Parker is the Support 5 Manager. I presume he was told to put it in his witness 6 statement. 7 Q. Told by who? 8 A. I would say the Post Office lawyers. 9 Q. So who was this discussion between, about the 10 suitability of Gareth Jenkins as a witness? 11 A. I'm not totally sure. 12 Q. What was the outcome of the discussion, that he should 13 give evidence or shouldn't give evidence? 14 A. I always think Mr Jenkins should give evidence. He 15 knows -- 16 Q. Do you know why he didn't give evidence in the Group 17 Litigation? 18 A. No, you would have to ask. 19 Q. Did you contribute to the drafting of Mr Parker's 20 witness statements? 21 A. Yes. 22 Q. Why did you contribute to the drafting of Mr Parker's 23 witness statements to the High Court? 24 A. Because he asked me to. 25 Q. Did you provide comments or instructions to the Post 98 1 Office solicitors on the evidence that Richard Roll, 2 a whistle-blower, had given about the facility of 3 Fujitsu to have remote access? 4 A. I almost certainly provided comments. I think 5 I provided comments on several witness statements. 6 Q. Why was it then that Mr Parker was the witness who was 7 selected to give evidence? 8 A. I expect he provided comments as well. 9 Q. Can we look, please, at FUJ00083835. This is the first 10 of Mr Parker's witness statements to the High Court. 11 You'll see there are some uncontroversial introductory 12 remarks and, on page 2, at paragraph 8, he begins 13 a section of his statement commenting on Mr Richard 14 Roll's witness statement, dated 11 July 2016. 15 Paragraph 9., a further description of essentially 16 the difference between Legacy and Horizon Online. 17 Then paragraph 10, please, comments on Mr Roll's 18 work. 19 Then, over the page, please, to paragraph 11: 20 "In his statement Mr Roll suggests that there were 21 frequent instances of software problems in Horizon that 22 had an impact on branch transaction data and which 23 Fujitsu resolved 'remotely' (ie not in a branch), not 24 merely by changing software but also by frequently 25 changing branch transaction data (by injecting new 99 1 transaction data and by editing or deleting existing 2 transaction data), without informing branches that such 3 actions were being taken ... those suggestions are 4 incorrect and Mr Roll's account ... is inaccurate and 5 misleading." 6 Did you contribute to the drafting of that 7 paragraph? 8 A. No, but I agree with it. 9 Q. You agree with what is said? 10 A. I agree that we didn't make frequent changes. I went 11 through the ACPs and OCRs that we used to record such 12 things and I think in 10 years I've found evidence of 13 28 financial remote changes, and I also disagree that we 14 didn't tell the subpostmasters. I've only ever seen one 15 PEAK where I think that that was mentioned. 16 Q. Forward to paragraph 16, please. Mr Parker says: 17 "It was (and is) theoretically possible for there to 18 be a software problem which could cause a financial 19 impact in branches, but this was (and is) extremely rare 20 and Horizon's countermeasures were (and are) very likely 21 to pick such matters up. In my experience, these 22 problems have always represented a very small proportion 23 of issues which led to software changes and a very small 24 proportion of the overall issues dealt with by the SSC." 25 Did you contribute to the drafting of that 100 1 paragraph? 2 A. No. 3 Q. Was it only theoretically possible for software problems 4 to cause financial impacts in branches? 5 A. No, we had evidence through the PEAKs. 6 Q. So it wasn't just theoretically possible, it had 7 actually happened? 8 A. Correct. 9 Q. Page 4, paragraph 18, please: 10 "In Legacy Horizon it was possible for the data in 11 a particular counter in a branch to become inconsistent 12 with replicated copies, and Mr Roll appears to be 13 describing this in paragraph 17 of his statement. In 14 that situation there could be remote management by 15 Fujitsu to correct the problem, but branch transaction 16 data was not changed in any way. As explained ... 17 below, the workaround involved replicating the correct 18 data from the counter in the affected branch or from the 19 data centre copy." 20 Did you contribute to the drafting of that 21 paragraph? 22 A. No. 23 Q. Is what is said in the second sentence, "there could be 24 remote management by Fujitsu but branch transaction data 25 was not changed in any way", accurate or inaccurate? 101 1 A. For -- I think we're talking about marooned transactions 2 here, which was what we covered in my witness statement 3 3, and you would not change the data that you recover 4 from a marooned transaction, apart from making it so it 5 doesn't clash with any new transactions entered. 6 Q. Paragraph 19, please: 7 "The suggestion that Fujitsu edited or deleted 8 transaction data is not correct. In Legacy Horizon it 9 was not possible to delete or edit messages that had 10 been committed to the message store." 11 Did you contribute to the drafting of that 12 paragraph? 13 A. I don't believe so, no. 14 Q. Is what is said in the first sentence there accurate or 15 inaccurate? 16 A. That is accurate. Once it's been inserted and 17 replicated, then you don't -- cannot edit. You only 18 add. 19 Q. At paragraph 20, please -- in fact, we needn't go on to 20 paragraph 20. 21 Do you know that Mr Parker made a second witness 22 statement in which he climbed down from some of the 23 things that he said in his first? 24 A. I believe he did make a second, I can't remember what 25 was in it. 102 1 Q. Well, in particular -- given the constraints of time, 2 I'm not going to go through it all with you -- he says 3 that in his witness statement, Mr Roll describes 4 a process by which transactions could be inserted via 5 an individual branch counter by using the correspondence 6 server to piggyback through the gateway. That's 7 a correct description of a form of remote access, isn't 8 it? 9 A. Yes, because, once you've inserted the message into the 10 correspondence server, it will be replicated down to the 11 counter. 12 Q. Do you know why that did not appear in Mr Parker's 13 evidence to the court in his first witness statement? 14 A. No, I don't. 15 Q. Were you providing instructions and information to 16 Mr Parker on which he made his witness statements 1 and 17 2? 18 A. I definitely commented. He emailed me and asked me for 19 comments, so I definitely commented. I wouldn't say 20 I provided instructions. I would never instruct him. 21 Q. Do you know why Mr Parker did not mention this form of 22 remote access in his first witness statement? 23 A. No, I don't. 24 Q. Was that the subject of discussion with you? 25 A. I don't know, actually, whether I commented on it during 103 1 one of my comments on his witness statement but, I'm 2 sorry, I could not tell you. 3 Q. Do you know why a witness statement that was addressing 4 the topic of remote access did not volunteer this form 5 of remote access that was available to Fujitsu at all? 6 A. No, I don't know. 7 MR BEER: Yes, thank you. 8 Sir, those are the questions that I would wish to 9 ask. 10 SIR WYN WILLIAMS: Are there questions from Core 11 Participants? 12 MR BEER: I'm just looking for a third shake of the head and 13 a fourth. 14 No. No, there aren't, sir. 15 SIR WYN WILLIAMS: So that completes the questioning? 16 MR BEER: Yes, it does, sir. 17 SIR WYN WILLIAMS: Thank you, Mr Simpkins, for returning to 18 the Inquiry, for providing two further witness 19 statements and for answering Mr Beer's questions this 20 morning and early this afternoon. I'm grateful to you. 21 THE WITNESS: Thank you, sir. 22 MR BEER: Sir, might we adjourn until 2.05, please. 23 SIR WYN WILLIAMS: Certainly, yes. 24 MR BEER: Thank you very much. 25 (1.05 pm) 104 1 (The Short Adjournment) 2 (2.08 pm) 3 MS PRICE: Good afternoon, sir, can you see and hear us? 4 SIR WYN WILLIAMS: Yes, thank you very much. 5 MS PRICE: May we please call Mr Barnes. 6 GERALD JAMES BARNES (sworn) 7 Questioned by MS PRICE 8 MS PRICE: Could you confirm your full name, please, 9 Mr Barnes? 10 A. Mr Gerald James Barnes. 11 Q. Thank you for coming to the Inquiry to assist it in its 12 work. As you know, I will be asking you questions on 13 behalf of the Inquiry. You should have in front of you 14 hard copies of two witness statements in your name, in 15 a bundle. The first is at tab A of that bundle and is 16 dated 30 August 2023. If you could turn, please, to 17 page 23 of that, please. 18 A. Right, yes. 19 Q. Do you have a copy with a visible signature? 20 A. Yes, I do, yes. 21 Q. Is that your signature? 22 A. It is my signature, yes. 23 Q. The second statement is at tab A2 of that bundle and is 24 dated 19 December 2023. If you could turn to page 13 of 25 that statement, please. 105 1 A. Right, yes. 2 Q. Is there also a visible signature on that copy? 3 A. There is, yes. 4 Q. Is that your signature? 5 A. It is my signature, yes. 6 Q. Are the contents of your statements true to the best of 7 your knowledge and belief? 8 A. Yes, they are, yes. 9 Q. For the purposes of the transcript, the references for 10 Mr Barnes' first statement is WITN09870100 and the 11 reference for the second statement is WITN09870200. 12 Mr Barnes, I will not be asking you about every 13 aspect of the statements that you have provided, which 14 will be provided and published on the Inquiry website in 15 due course. I will instead be asking about certain 16 specific issues which are addressed in them. 17 Starting, please, with the relevant roles you have 18 held with Fujitsu Services Limited over the years you 19 have spent in its employment, in broad terms, you have 20 been a software developer with Fujitsu since 1998; is 21 that correct? 22 A. That is correct, yes. 23 Q. You remain employed by Fujitsu? 24 A. That is correct, yes. 25 Q. You explain in your first statement that your first job 106 1 with Fujitsu involved looking after a database of 2 reports produced by Post Office clerks; is that right? 3 A. That's right, yes. 4 Q. You then became involved in supporting the Electronic 5 Point of Sale Service, or EPOSS, software for 6 transacting at the counter and balancing, as well as 7 looking after related reports? 8 A. That's correct, yes. 9 Q. Can you recall the year in which you became involved in 10 supporting EPOSS software? 11 A. Not exactly, no. But pretty soon, I think, I got to 12 grips with the reports and got that all under control, 13 and the designer in the team thought, oh, probably about 14 time to give me some more work to do too, because I kept 15 doing the reports, I kept that all under control, but 16 I sort of alternated it and got it moderately 17 streamlined, so I had time to do other work and I think 18 that's when I started looking at other things. 19 Q. So was it within the first year that you joined Fujitsu, 20 if you joined in 1998? 21 A. That's pretty -- moderately soon, I would say. I just 22 can't remember exact dates. 23 Q. Whilst you were in this role, you did an evening class 24 in bookkeeping; is that right? 25 A. Ah, yes, that's when I started looking at this balancing 107 1 and I found that very, very interesting so, yes, in my 2 own time I got some accounting qualifications just 3 because I found so it interesting, that was why. 4 Q. You give some examples of software which you developed 5 at paragraph 6 of your first statement. Could we have 6 that on screen, please. It is page 2 of WITN09870100. 7 At paragraph 6, you say: 8 "I remember writing a component called 'Operation 9 Launch'" -- 10 A. Yeah, there was quite a few things I did but that was 11 certainly one of them. That was when we were starting 12 looking at sales with debit cards and credit cards, yes. 13 That was a part of that project. 14 Q. You say: 15 "[It was] to facilitate the use of [those] debit and 16 credit cards" -- 17 A. That's right, yes, yes. 18 Q. -- "which was being introduced in the earlier version of 19 the Horizon system [legacy Horizon]." 20 A. That's right, yes, that's correct, yes. 21 Q. You go on to give another example of software you wrote 22 at paragraph 6, and you say this: 23 "I also remember writing the migration software that 24 enabled a counter to transition from using Escher's 25 Riposte software platform to the new system (known as 108 1 'HNG-X' or 'Horizon Online'). Because of this piece of 2 work, I believe I was the last member of the EPOSS 3 Riposte team, which was a large team during the time of 4 Legacy Horizon." 5 Just pausing there, you say it was a large team. 6 Can you remember how large your team was? 7 A. Not precisely, but 10 or 20, maybe. I couldn't give you 8 an exact figure. 9 Q. You deal at paragraph 7 of your statement with the 10 circumstances in which you moved to the Audit Team and 11 you say this: 12 "In 2009 or thereabouts, whilst supporting the 13 migration software for the remaining counters to 14 transition to HNG-X, I also started looking at the audit 15 system in HNG-X, which was a completely new area for me. 16 It was around this time that I then joined the Audit 17 Team. I recall that there was already an audit system 18 in Legacy Horizon for Riposte that I knew little about 19 then. When I joined the team, this audit system was 20 being rewritten as part of the transition to HNG-X. For 21 this reason, I have limited experience and knowledge 22 regarding the systems and processes relating to audit 23 and ARQs in relation to Legacy Horizon." 24 It's right, isn't it, that the Audit Team was and 25 remains responsible for providing to the Post Office, 109 1 when requested to do so, audit data retrieved from the 2 audit servers, for the purposes of Post Office 3 investigation of and criminal and civil or disciplinary 4 action against subpostmasters, their assistants and 5 managers, and those employed by the Post Office; is that 6 right? 7 A. That's partly true but you can -- they have queries for 8 very many other reasons why you want to look at historic 9 data but, certainly, that's one of the reasons, yes. 10 Q. You have remained in the Audit Team since you joined in 11 around 2009; is that right? 12 A. That's right, I've done a few other things as well, when 13 things have been slack but I've always been responsible 14 for the audit software and still am, in fact. 15 Q. Turning please to the point at which you began 16 supporting the EPOSS software, when you took up this 17 role, were you aware that an EPOSS taskforce had been 18 established in August 1998 to address the escalating 19 number of PinICLs being raised which led to the 20 taskforce reporting significant deficiencies in the 21 EPOSS product, its code and its design? 22 A. No, in fact -- I certainly was not aware of that then 23 and I'm not even sure I'm aware of it until you've just 24 told me. 25 Q. Do you recall the rollout of Legacy Horizon? 110 1 A. Not the rollout, I don't think, because it would have 2 already started rolling out before I joined but there 3 were numerous further releases, improved releases, all 4 the time. 5 Q. Do you recall ever being made aware of an Acceptance 6 Incident in around July 1999 which related to accounts 7 not balancing sufficiently or at all? 8 A. I'm aware of quite a few non-balancing issues. Can you 9 be more specific in giving me some -- a pointer to this 10 particular one? Is there a page reference? 11 Q. At this stage I'm referring to an Acceptance Incident -- 12 A. Right. 13 Q. -- in the course of the negotiation of the rollout of 14 Legacy Horizon. Do you remember being told about 15 an Acceptance Incident that related to balancing? 16 A. Not specifically. I might have been but I can't be 17 specific. 18 Q. You have fairly recently been provided by the Inquiry 19 with a number of documents relating to your involvement 20 in reported issues with Legacy Horizon. I'd like to ask 21 you, please, about a number of those documents. Could 22 we have on screen, please, document reference 23 POL00028747. This is a PEAK system management system 24 log -- "Peak Incident Management System" log. The call 25 reference is at the top left of the document, PC0059497, 111 1 and at the top right we see you identified as the call 2 logger. At the risk of stating the obvious, does that 3 mean that you logged the call to which this log relates? 4 A. I think it means I cloned the call -- well, yes I did, 5 but it's a call type cloned call. I assume, although 6 I can't remember for sure, that there must have been 7 an existing PEAK which I then cloned for some reason and 8 that's why I became the call logger. 9 Q. Can you help, please, with what a cloned call is -- 10 A. Oh, it's just -- you have PEAKs and, for some reason or 11 other, you might want to have a copy so that one is used 12 for one purpose in resolving issues and another copy is 13 used for another issue in resolving issues. So you 14 clone it so you've got two of them, and then one might 15 take one path and the other might take another path. 16 For example, if you've got two releases going on and 17 you want an urgent fix to go out to live in the first 18 but you've got to catch it up in software being 19 developed for the follow-on release, then you'd need 20 two: one for the live and one for the follow-on release. 21 That's just one example. I mean, there are others. 22 Q. The first entry on the log shows the call was made on 23 20 November 2000 at 13.19 and the entry at 13.20 says 24 this: 25 "Receipts vs payments difference at 145004 for CAP 112 1 34. This is not a migration issue. This outlet has no 2 other open calls on PowerHelp. Please investigate and 3 confirm if this is a CI3 or CI4 office. If this is 4 a CI4 office this may be a new problem." 5 Was this your entry or not? 6 A. No, no, that's -- no, that's customer call. That 7 wouldn't have been something I added. That would be -- 8 no, no, that would be someone else's entry. My entries 9 would always have my name -- where it's got 10 "User:_Customer call_", my entries would always be 11 "User: Gerald Barnes". 12 Q. Going over the page, please. Towards the bottom of the 13 page there is an entry dated 8 December 2000, timed at 14 12.33. If we could zoom in a little on that, please. 15 Here is an example of just that, "User: Gerald Barnes"; 16 so is this is an entry made by you? 17 A. Yes, this would definitely be, yes. 18 Q. It says: 19 "New evidence added -- Messages produced when stock 20 unit OOH was rolled. 21 "F) Response:" 22 Can you help with "OOH"? 23 A. Oh, that's just the name of the stock unit so all the 24 stock units are given different names and that's just 25 one of the names. It could be anything, really. Any -- 113 1 it's just the same of the stock unit. 2 So I mean, in an office -- well, if it's a very 3 small office, you might only have one stock unit but you 4 could have more than one or many. In a very big office, 5 you'd have many stock units. 6 Q. You say here: 7 "This is another case of transactions being dropped. 8 At CI3_2R this happens with no error logged. At CI4L1 9 and above, it is often the case that an error will be 10 reported to the user in such cases." 11 Can you help with what you were referring to by 12 CI3_2R? 13 A. Right, these are just the names of the releases. So, 14 well, I mean, I can't remember in detail but in general 15 each release rolled out of the EPOSS software would have 16 some reference number and, although I can't remember 17 that far back, these must have been the references to 18 the various releases. 19 Q. You go on: 20 "I will have another look at M1 rollover and see if 21 any further improvements can be made in error trapping 22 to catch other Riposte Errors." 23 You refer in this entry to this being another case 24 of transactions being dropped. Is it fair to say, 25 therefore, that this was an issue which you knew was not 114 1 an isolated one? 2 A. Well, we're going back in time a long way but I didn't 3 write all this cash account software myself originally 4 but I did spend a lot of time looking at the code and 5 looking at PEAKs and trying to improve it. So it sounds 6 like I could see another place where it could be 7 improved, in this case to try to make the error handling 8 better than it was before. 9 Q. But this issue of transactions being dropped, you're 10 referring to that as being another case. 11 A. Yes. 12 Q. So transactions being dropped, this isn't, it seems, 13 an isolated case of that? 14 A. Yes, I -- from what I wrote, that must be -- it must be 15 right, yes. I can't specifically remember that far back 16 but I've written what I've written so, yes, it must be 17 the case that I've spotted this before. 18 Q. Is it right, on the face of your entry here, that this 19 was a problem caused by a Riposte error? 20 A. Oh, that's right. That's right. So the very -- the 21 basic Riposte errors should -- if they go wrong, they 22 should -- they have their own error mechanism, which you 23 should be able to catch. And I think what I'm saying is 24 that the errors weren't being caught properly, that's 25 what I'm saying. So they could have failed and not been 115 1 noticed. Though I've subsequently discovered, actually, 2 you tend to get what's called an event written to the 3 event log always. So one of these Riposte calls 4 failing, although it might not be caught in the 5 software, typically it would go to the Windows event log 6 and would get something, a red event there. 7 So, after the event, you would be able to spot these 8 things by checking the Windows event log but the 9 software itself did not catch the errors and, in my 10 view, that's much better, if the software itself catches 11 the errors and reports back. 12 The ideal case, if it was really written perfectly 13 from the word go, anything goes wrong, when the 14 postmaster rolls over the stock unit, you should have 15 a message it's logged somewhere in the event log -- 16 doesn't matter where, somewhere -- and a clear message 17 reported to the postmaster "This has gone wrong, please 18 contact the Helpdesk". 19 That's how ideally it should all work but, at this 20 time, it wasn't like that. 21 Q. Being specific, this error, where it occurred at CI3_2R 22 did not result in an error message coming up for the 23 user. That's what you're suggesting? 24 A. That's what it says so I suppose that's right, yes. 25 I mean, I can't really remember that far back in detail 116 1 but that's what I've written. 2 Q. So if it happened and appeared as a misbalance to the 3 user, is it right that it would require further 4 investigation of the message store or, depending on the 5 timings of the investigation, the audit data, to explore 6 whether an error in Horizon was to blame? 7 A. That's right, yes. That's right. 8 Q. What if the user did not report the issue and there was 9 therefore no investigation? 10 A. If the user didn't report -- well, then it goes 11 unnoticed but there will be some sort of error if the -- 12 well, hmm, I suppose it's always possible if nothing is 13 noticed, I suppose. But, yes, unless the user reports 14 something, then we're not going to know about it, 15 I would say. 16 Q. You say in your entry that at CI4L1 and above, it is 17 often the case that an error will be reported to the 18 user -- 19 A. It looks like things were improved then, yes. 20 Q. You say "often" but not always. 21 A. Yes, that's right. That's right. I think that's 22 probably right. 23 Q. In the context of a balancing problem, a failure in 24 error reporting is a significant problem, isn't it? 25 A. Definitely. I would say absolutely, yes. Yeah, 117 1 definitely. 2 Q. Going over the page, please, there is another entry made 3 by you, dated 11 December, which is three entries down. 4 If we can zoom in a little, please. You appear to 5 record that a fix was implemented. Was this a fix that 6 was carried out by you, can you say? 7 A. Doesn't say explicitly there. It's the sort of thing 8 I'm likely to be involved with but I can't say for sure 9 one way or -- it might have been another developer. It 10 is not explicit, is it? I can't remember this event 11 well enough to be able to be assertive in my response 12 there, other than what's written. So might have been 13 me, might have been another developer. 14 Q. Two entries down, on 18 December, we can see an entry 15 from Clifford Sawdy. He notes that there is: 16 "... no specific test that can be performed to prove 17 a fix for the original problem regarding missing 18 transactions." 19 Then there is an entry on 17 January 2001, which 20 says this: 21 "We've run through complete M1 test cycles, and 22 subsequent stock unit rollover and cash account testing 23 as described above by Cliff, and have been unable to 24 reproduce this error. Suggest this is now closed." 25 Then, finally, an entry on 18 January, starting at 118 1 the bottom of the page, going over to the next, please, 2 and it says: 3 "Closing call as fixed at future release [date] PM 4 has not been informed." 5 Why would the postmaster not be informed about the 6 outcome of the investigation? 7 A. I couldn't answer that question. I was fourth line 8 support. This would be some higher up level of support. 9 I don't know the answer to that question. 10 Q. There are no further entries on this log to evidence any 11 further check to ensure the problem, which had not been 12 reproduced by testing by this point, would not be 13 an issue in future releases. We can't see any evidence 14 of that, can we? 15 A. That's right. Well, I think -- well, it's a long time 16 ago but my guess is that it was some sort of 17 intermittent problem and, therefore, very difficult to 18 test. You can't really, if it's intermittent failure, 19 you can't really. It's very difficult. The best they 20 could do is regression test everything and, if someone, 21 which might have been me, has just simply improved the 22 error handling in some area, all that means is that, 23 next time this intermittent problem comes up, you'd have 24 more evidence than before the improvement in error 25 handling. 119 1 Q. At the time, did you recognise the implications of 2 an error in Riposte, or otherwise, causing a discrepancy 3 in the accounts of a branch without the user in branch 4 being aware of that error? 5 A. I don't think I would have thought about it. These were 6 just technical issues to me, which I did my very, very 7 best to fix, but I don't think my mind would go in that 8 direction, really. 9 Q. Could we have on screen, please, document reference 10 POL00028750. This is a PEAK which appears to relate to 11 the same call on 20 November 2000, which was the subject 12 of the last PEAK we've just looked at but it has 13 an extra entry from you, and we don't have the reference 14 to a cloned call. So is this is an example of another 15 document that records the call that's used for 16 a different -- 17 A. Well, I don't know for sure but possibly this is the 18 original call and the call you showed me first of all is 19 cloned from it. That's possible. I don't know for sure 20 to without checking it carefully. 21 Q. Could we look, please, to the bottom of page 2. We see 22 there the same entry we've just looked at. We don't 23 need to zoom in on that but just to show you that it is 24 the same entry there on 8 December. Then, over the 25 page, please. The top entry there, also 8th December -- 120 1 if we could just zoom in a little, please -- we see at 2 the very top of the page: 3 "Call PC0058161 cloned to new call PC0059497." 4 Then this entry from you here at 12.38, and: 5 "As already stated ... this is a case of Riposte 6 System calls failing with no error being logged. At 7 CI4L1 things are much better. The call has been cloned 8 ... to improve even further still the logging of Riposte 9 System call errors in stock unit rollover." 10 You may not be able to help at this remove but can 11 you help with what you meant by "At CI4L1 things are 12 much better"? 13 A. Well, I -- I can't remember the details but, presumably, 14 I wrote that because I was aware that a lot of fixes 15 were going into CI4L1 but my memory isn't that good. 16 I can read what I wrote but that's all I can imagine was 17 the case, that, in general, we'd done more PEAK fixes 18 for CI4L1, the development team in general and, no 19 doubt, I helped in that too. So we thought that things 20 would be better in that release. 21 Q. It does not sound from this entry as though CI4L1 was 22 a complete fix, does it? 23 A. Oh no, no. I mean, you could never get every single bug 24 from a system. That's just -- you do your best but it's 25 just impossible. There's always bound to be some bugs 121 1 in systems. 2 Q. Turning, please, to your knowledge of later issues with 3 Legacy Horizon, could we have on screen, please, 4 FUJ00090436. You refer in your statement at paragraph 6 5 to your involvement in Operation Launch and we've looked 6 at that reference, which you say related to facilitating 7 the use of credit and debit cards in relation to Legacy 8 Horizon. This appears to be a report relating to 9 Operation Launch. The release referenced is BI3(S70). 10 We can see you're listed as the originator and 11 department. Were you the author of this document? 12 A. Yes, I think so, that's when you -- yes, I think so, 13 yes. 14 Q. The document is dated, looking to the top right-hand 15 corner, 12 January 2005. Could we turn, please, to 16 page 6 of this document. Under the heading 17 "Non-Functional Tests" and the subheading "Performance" 18 we have this: 19 "Pool paged bytes for both Desktop and Riposte were 20 monitored with Performance Monitor for 7212 cycles of 21 the soak test ... which meant over 14,000 operations 22 were launched. No memory leakage was detected in the 23 Desktop at all -- for Riposte the figures are given in 24 the table below." 25 Then you set out some figures. You say: 122 1 "This is much more likely to be a problem with 2 Riposte than with Operation Launch since Operation 3 Launch shares its memory with Desktop." 4 Then, over the page, please. There is summary of 5 problems found: 6 "The only possible problem found was that Riposte 7 may have a memory leak. It is considered beyond the 8 scope of this module test to progress this further." 9 Can you help, please, with what you mean by 10 a "memory leak"? 11 A. Right, well, this is something you've got to look out in 12 software development. Sometimes you allocate memory 13 dynamically, for some temporary period of time and then 14 you've always got to be sure to delete the memory block 15 when you've finished with it. If you don't, you can end 16 up with a memory leak, where your program just starts 17 using more and more memory until eventually you run out 18 of memory. So you've always got -- if you're testing 19 something thoroughly, you should always test for memory 20 leaks to make sure your new component doesn't have any 21 memory leaks. 22 Q. What were the implications of a memory leak for the 23 functioning of Riposte? 24 A. Oh, well, I don't think the figures were that big 25 a memory leak. So as long as it's small, you can get 123 1 away with it. It's only if you have a big memory leak 2 that you have real serious issues. As long as it's just 3 some small problem, then you can get away with that. 4 Q. Could we have on screen, please, FUJ00154684. This is 5 a PEAK log relating to a call from the National Business 6 Support Centre on 20 December 2007. The log reference 7 is PC0152376. You have dealt with this PEAK at 8 paragraphs 29 to 33 of your first statement. About 9 halfway down the page, we can see a summary of the issue 10 being raised. Starting with "Ibrahim": 11 "Ibrahim from the NBSC has asked that an issue be 12 investigated by our software team regarding 13 discrepancies still showing when the MIS stock unit is 14 rolled to clear the local suspense account." 15 Then under, "Incident History", there's some more 16 detail. It says: 17 "On Wednesday 12/12 the BM stock unit had a gain of 18 £465.73. As this stock unit rolled over it was forced 19 to clear local suspense £1,083.76. The gain of £465.73 20 did not go to local suspense and is not included in the 21 £1,083.76. 22 "This was not the last stock unit to roll over. The 23 last stock unit to roll over was MIS at 10.20 on 13/12. 24 This stock unit had no discrepancies. MIS is 25 a correction stock unit and was not inactive as it is 124 1 rolled over every BP. 2 "The suspense account and final balances corroborate 3 the above as the office has sent us copies. 4 "The trading statement agrees with the suspense 5 account and that BM stock cleared suspense but did not 6 send its gain to suspense. The trading position line 7 should always show zero. Under the BM stock column it 8 shows £465.73. 9 "I have had a trial done on BM stock to see if this 10 is showing the £465.73 but it is not." 11 So the problem being reported was one of 12 discrepancies in the account; is that right? 13 A. Yes, that's right. That's right, yes. 14 Q. If we could go to page 3 of this document, please, there 15 is an entry made by you on 2 January 2008, that second 16 entry down, in which you say: 17 "The fact that EPOSS code is not resilient to errors 18 is endemic. There seems little point in fixing it in 19 this one particular case because there will be many 20 others to catch you out. For example when I tried to 21 balance with CABSProcess running I found that declaring 22 cash failed with the same sort of error message!" 23 Pausing there, can you explain, please, the role 24 which EPOSS code played in relation to the error which 25 was operating in this case? 125 1 A. Well, I mean, the EPOSS -- that is what the stock 2 balancing is -- it is the EPOSS code, is -- stock 3 balancing code is part of the EPOSS code but the EPOSS 4 code is more general than that. There's lots of EPOSS 5 code. For example, just selling a stamp would be EPOSS 6 code but also, more specifically, stock balancing would 7 be part of the EPOSS code. 8 Q. Which errors was EPOSS code not resilient to? 9 A. Well, it's -- we just spotted cases where the error 10 handling was not as good as it could have been, which we 11 tried to eliminate over the years. So sometimes calls 12 to write out a message would fail silently, though as 13 I mentioned before, though silently to the code, you 14 always get a red event written into the Windows event 15 log, so you can -- so the postmaster wouldn't be 16 directly aware of the failure but analysis of the logs 17 after the event would show the problem. 18 In my opinion, it would be far better if, when 19 something like this went wrong, immediately the software 20 should abort and the postmaster should just be told 21 "An error has occurred, please contact the Helpdesk", or 22 something like that. So the error handling wasn't as 23 good as it could have been if designed properly from the 24 start, but that's not to say that the evidence wasn't 25 there to spot the problem after the event because we get 126 1 information in the Windows event log, et cetera. 2 So what I'm saying is the error handling, in 3 an ideal world, could have been done much better but, 4 nevertheless, it's not to say that you can't detect the 5 problem, because you can, and -- 6 Q. Apologies. That is what you're referring to, is it, 7 when you say that the code was not resilient to the 8 errors, to the error handling process? 9 A. That's right, yes. So it's just not as good as it could 10 have been: ideal behaviour, any problem, log it, abort, 11 just say to the postmaster "Please contact the 12 Helpdesk". That would be the ideal error handling in my 13 view. 14 Q. The reason for this, was this because there were 15 deficiencies in EPOSS code itself? 16 A. Well, in the error handling. I mean, I thought the 17 EPOSS code was quite clever, really, but in the error 18 handling, it wasn't done as well as it could have been 19 done, had the time been taken to do so. But the code 20 itself -- in programming you have what's called the 21 happy path. The happy path is when everything is being 22 done well. In the happy path there's no problems. 23 Q. Was this view, the fact that the EPOSS code was not 24 resilient to errors was endemic, a view that was held 25 within your team at the time? 127 1 A. Well, I only spoke to a few colleagues about the issue. 2 I could give you some hearsay quotes, if you like, but 3 I can't give names, I don't think. 4 Q. Were there others who shared your view? 5 A. Well, the people I talked to didn't seem to think that 6 way. For example, one colleague said, "Well, you've got 7 to assume all this fundamental stuff works, you've just 8 got to assume that". Another colleague said, "Well, 9 when it's all developed in the first place, it was 10 assumed that all the error handling would be 11 automatically added later". 12 So the two colleagues I informally mentioned this to 13 didn't seem to quite share my views, to be honest. But 14 that's not -- I didn't mention it to everyone in the 15 entire team, though. So ... 16 Q. In your entry, you gave an example of trying to balance 17 with CABSProcess running, and declaring cash failing 18 with the same sort of error message. Can you explain 19 what the CABSProcess was, please? 20 A. Yes, well, it's -- it was just a piece of software run 21 each evening about 7.00, which just -- you have end of 22 day markers which jot the -- divide each day and it just 23 summarises all the transactions that go on in the day in 24 some way. I can't remember the details beyond that but 25 it's just a summary of transactions that occur each day, 128 1 around about seven o'clock every evening. 2 Q. You describe in your first statement at paragraph 27 3 that an issue relating to the CABSProcess could cause 4 potentially incorrect data to be presented to the audit 5 system. Is that what happened here? 6 A. Yes, that's right. I mean that's right. So because of 7 it, the messages logged are incomplete. Yes. Nothing 8 is wrong with the audit system itself but the data to be 9 presented to it later would be incomplete. 10 Q. You go on in this entry in your PEAK to say this: 11 "It may be worth passing on the general message to 12 the HNG-X team that, in many cases code should always 13 try and exit gracefully after an error and not just 14 blunder on regardless. 15 "This is a perfect example of why. Had the 16 balancing code exited gracefully then if the user had 17 tried again after CABSProcess had finished working then 18 all would have been well." 19 Was the effect of this, the code not exiting 20 gracefully, that which to which you refer at 21 paragraphs 29 and 31 of your first statement, that the 22 failure is silent. 23 A. That's right, yes. Well, relative -- well, silent to 24 the postmaster. As I say, information is available in 25 the event log. It would be available to 129 1 a diagnostician, looking at it, but silent to the 2 postmaster. 3 Q. Could we have paragraph 31 of Mr Barnes' first statement 4 on screen, please. It is page 15 of WITN09870100. You 5 explain the silent failure point in this way, at 6 paragraph 31: 7 "The fact that the failure was silent was really bad 8 error handling. Good programming practices would be to 9 abort (ie for the code to stop running) with a clear 10 error message. It is better to produce no results than 11 incorrect results, and good error handling should be 12 coded from the start. However, my understanding is that 13 in PEAK PC0152376, an error was written to the audit log 14 and then processing continued, so although the operator 15 at the Post Office branch would not know anything had 16 gone wrong, a detailed analysis of the audit log after 17 the event would have revealed the problem." 18 This is substantially the same point, is it not, as 19 the point which arose in the context of the Riposte 20 error under CI3_2R in 2000, the lack of an error 21 message, meaning that the user is not alerted to 22 an error in the system having occurred. Would you agree 23 that it is substantially the same problem? 24 A. Yes, that's right, yes. It's the same sort of thing. 25 Q. It is arising now in 2007, into 2008. Could we have 130 1 back on screen, please, FUJ00154684, page 3, please. 2 You proposed a fix to the problem on 2 January 2008, and 3 you explained it in this way, starting 4 paragraphs down 4 in your entry: 5 "For the time being I propose a much cheaper 6 solution than rewriting a lot of EPOSS error handling. 7 "The problem is that because of a previous PEAK ... 8 CABSProcess writes out messages atomically. It does 9 a StartTransaction quite early on (which creates the 10 lock), then initiates writing lots of transactions with 11 CreateMessage and persistent objects with PutObject, and 12 finally really writes them with a call to EndTransaction 13 (which ends the lock). If something else tries to write 14 a transaction whilst CABSProcess has things locked then 15 it will time out after 10 seconds. Hence if CABSProcess 16 takes more than 10 seconds to run, you could get this 17 sort of problem. In this case, CABSProcess took 33 18 seconds to run which gives a significant window of 19 opportunity for this sort of problem to occur. 20 I suggest addressing this matter directly by having 21 CABSProcess store all that it wants to write out to 22 a collection and then only really write it out at the 23 very end. In this way the system will be locked for 24 less than 10 seconds and there will be no possibility of 25 this sort of problem." 131 1 Then two-thirds of the way down the page, you deal 2 with the "Impact on User", and you say: 3 "Benefit of making a fix. 4 "It will no longer matter if CABSProcess is running 5 when the user tries to do many sorts of things, 6 balancing included. 7 "What does the user have to do to get this problem? 8 "Do anything which involves writing a transaction 9 whilst CABSProcess is running (after 19.00) when 10 CABSProcess has sufficient work to do so that it takes 11 more than 10 seconds to run (so probably on the larger 12 offices)." 13 So just to be clear, you were warning here that the 14 CABSProcess issue could impact upon balancing? 15 A. Well, if the postmaster is working after 7.00, yes, 16 that's right. Well, or to be more precise, he's working 17 through 7.00 because that's when this process ran. So 18 I suppose he'd have been all right if he started 19 balancing at 7.30, for example. 20 Q. You go on to covering the impact on operations and you 21 say: 22 "Benefit of fix that may not be visible to end user. 23 "Less support calls." 24 So, in summary, you thought the risks of running 25 a fix were outweighed by the benefits? 132 1 A. Yes, it was quite an easy fix, really. So I thought 2 quite safe. 3 Q. Under those "Risks": 4 "What live problems will there be if we do not issue 5 this fix? 6 "Problems will continue to occur if the counter is 7 being used whilst CABSProcess is running, in those cases 8 when it takes more than 10 seconds to run." 9 Referring to that risk again, in terms of 10 operations: 11 "Is this a high risk area in which changes have 12 caused problems in the past? 13 "Yes. However the fix proposed is self-contained 14 and is considered unlikely to cause any problems." 15 Going over the page, please, towards the bottom of 16 this page there is an entry from David Seddon, dated 17 10 January 2008. He says this: 18 "It has been decided that no fix will be carried out 19 for the time being given the rarity of the problem. 20 Should the problem become more prevalent then the need 21 for a fix will be reviewed once again. In the meantime 22 KEL dsed5628Q has been created to cover the problem. 23 "With regard to this instance of the problem we have 24 already passed details and corrective actions necessary 25 to Post Office Limited by means of a BIM issued by the 133 1 MSU ... Therefore no further action is necessary and 2 this call can simply be closed." 3 Should we take it from this that a decision was made 4 that, despite your recommendation, it was decided by 5 10 January 2008 that no wider fix would be implemented? 6 So we have this narrow fix, but it appears no wider fix 7 to the problem. 8 A. That's right. That is correct. 9 Q. Could we have on screen, please, FUJ00155261. This is 10 an email chain from September 2008. The first email in 11 the chain starts on page 2, could we turn to that, 12 please. Towards the bottom of the page, this is 13 an email from Gareth Jenkins to Roy Birkinshaw, copied 14 to you, Steve Evans, John Burton and Anne Chambers. It 15 is dated 4 September 2008 and reads as follows: 16 "Roy, 17 "As requested yesterday, I've had a look at the 18 relevant code and a chat with Gerald and I am satisfied 19 that the fix that Gerald has proposed for this PEAK is 20 low risk and should remove this particular cause of 21 timeouts. The actual PEAK is now closed, so I'm not 22 sure exactly what process should be followed, but 23 effectively what I think we need is for the PEAK to be 24 reopened and sent to RMF for further consideration in 25 light of recent investigations. 134 1 "Are you and Steve able to progress it from here?" 2 In the email above, Steve Evans asks you to "liaise 3 with Dave Seddon/Lionel to get this reopened and then 4 back to RMF". What was RMF? 5 A. Release Management Forum. 6 Q. Going back to the bottom of page 1 of this document, 7 please, this appears to lead to an email from John 8 Budworth to you, and a number of others, including Mik 9 Peach and Gareth Jenkins, and he says this: 10 "All 11 "PEAK 164429, (clone of 152376) has arrived in RMF. 12 At the moment this is the only PEAK in RMF. I'm not 13 sure why this has been revisited by CounterDev and 14 Gareth as we decided we were not going to fix this back 15 in January. 16 "Has something in live increased the problem or has 17 it beed [I think that should be 'been'] raised as 18 an issue by the customer or elsewhere? I don't know. 19 "Anyway, CABSProcess is start of LFS_COUNTER. I am 20 not expecting any other LFS change during Horizon but it 21 might be worth looking at LFS related PEAK 147179." 22 Then going to the email above that, please, we have 23 an email from Steve Evans: 24 "I note that Mik has replied, and yes this one has 25 become a higher priority with the customer. 135 1 "It's not related to PC147179, which I've actually 2 just returned 'no fault in product', so doesn't exist 3 any more. 4 "Gerald has requested a target of T86 and he has 5 gone off on leave until 23 September. Therefore a fix 6 will not be available before the 25th." 7 In this email, the customer, was that the Post 8 Office? 9 A. Yes, that's correct, yes. 10 Q. So was it your understanding that the Post Office were 11 at least by this point aware of the issue that had 12 arisen? 13 A. Where does it say, "The customer", exactly? 14 Q. The first line of that email, "I note that Mik has 15 replied and yes this one has become a higher priority 16 with the customer". 17 A. Yes, I suppose that must be the case, I suppose. It 18 must be. 19 Q. John Budworth's email reply is above that, and he says: 20 "Thanks all, 21 "I'll check RMF stack again tomorrow but nothing 22 other than this PEAK in there currently. I'll authorise 23 PEAK 164429 for T86 but would like to move this forward 24 sooner rather than later so test and deploy as early as 25 possible in October." 136 1 To the extent that you are able to recall, were you 2 involved in any discussions about the merits of a fix 3 between the 10 January 2008 decision recorded by David 4 Seddon in that original PEAK and the email from Gareth 5 Jenkins on 4 September 2008? 6 A. No, I was kept out of the loop completely. I'd have 7 been busy looking at other PEAKs, I expect so, no, 8 I wasn't aware of that. 9 Q. You explain at paragraph 32 of your statement, that is 10 your first statement, that, having reviewed the PEAK at 11 document reference FUJ00155366, you can see that a fix 12 was applied on 25 September 2008 and that you were 13 involved in applying that fix, is that right? 14 A. Yes, that's correct, from just my review of documents 15 recently, because of this -- my witness statement, yes. 16 Q. You also explain at paragraph 32 that you had some 17 involvement checking event logs in January 2009. Could 18 we have on screen, please, FUJ00154836. About halfway 19 down the page is an email from Penny Thomas to you and 20 Steven Meek, copied to Gareth Jenkins and Anne Chambers. 21 It is dated 31 December -- 22 Apologies, if we can scroll out, please, that bottom 23 email, 31 December 2008, please. The email is dated 24 31 December 2008. The subject line is "ARQs 499-509", 25 and then a reference "475329 -- LPD 19 January 2009". 137 1 Ms Thomas says: 2 "Hi Gerald 3 "Could you please check events for the following 4 ..." 5 Then giving that reference with the date range of 6 21 September '07 to 17 August '08. 7 "Many thanks 8 "Penny." 9 You then send an email in reply on 5 January 2009 10 a bit further up the page. Scrolling down, please, we 11 can zoom out and see the whole document. That's fine. 12 You appear in that email to attach some results. 13 "Hi Penny, 14 "I attach the results. 15 "Regards, 16 "Gerald ..." 17 Then at the top of the page there is an email from 18 Anne Chambers to Penny Thomas, copied to Gareth Jenkins 19 and to you. It says: 20 "475329 counter 3 Lock events 28 March 2008, 22.04, 21 checkpoints being written during Smartpost upgrade. 22 Just confirm no one logged in." 23 Does this description at the top help you at all to 24 say what you were checking events for, or not? 25 A. I can only remember the general process. Once I joined 138 1 the Audit Team and I think subsequently I didn't know 2 this at the time but, reading all the material for my 3 appearance today, I've discovered it, because of my old 4 statement, the error handling wasn't very good, a new 5 system was introduced where the event logs was always 6 checked before any spreadsheet -- any spreadsheets or 7 transactions were sent to the Audit Team, by the Audit 8 Team, to check that there were no suspicious events, 9 that occurred at the time of the transaction as reported 10 in the ARQ. 11 So a database of all the event logs -- all the event 12 logs were extracted, they were stored in some sort of 13 database and I was partly involved in, when requested, 14 getting events back for a given date range. 15 So what Anne is saying is she's looked at these 16 events, and decided that, other than these three lock 17 events, there was nothing suspicious in the event logs 18 that I returned to her. She's saying those are the 19 suspicious ones and, moreover, she's saying, as long as 20 no one was logged in at the time, they don't matter. 21 Q. Can you say whether this check related to the 22 CABSProcess issue we've been looking at or not? 23 A. I think -- well, as I say, because I have reviewed 24 the -- what went on from the information presented to me 25 before my appearance today, I can say yes, because of my 139 1 comments -- not necessarily the CABS issue, just because 2 I said in general -- these Riposte errors can be silent 3 to the postmasters in general, well, the CABSProcess as 4 well, but in general it is the case, that the policy was 5 adopted of always checking all the event logs for any 6 ARQ evidence presented, so that we can be -- they could 7 be more certain that nothing like that had gone on. 8 So that was a new -- because of my -- it now 9 transpires from what I've read, because of my comments, 10 this new process was adopted. But it was all -- 11 never -- at the time I knew nothing about it. It was 12 all completely silent to me but I can see that is the 13 case from what I've read subsequently. Just -- well, in 14 recent weeks. 15 Q. Could we have on screen, please, FUJ00155402. About 16 halfway down the page is an email from Penny Thomas to 17 Steve Evans, among others. It is dated 8 January 2009, 18 so three days after you replied with results in the 19 email chain we've just looked at. Ms Thomas says this, 20 under a subject "Audit Issue": 21 "As a result of our meeting today the following 22 actions have been agreed: 23 "1. We will event check all transaction data 24 supplied to POL where that data falls between May '07 25 and November '08. 140 1 "2. The check will focus on events where the 2 CABSProcess has produced a lock from 1900 hours to 1910 3 local time. 4 "3. Penny to provide a list of 195 outlets with 5 time frame. 6 "4. Alan to provide query. 7 "5. Gerald to run the event check through the 8 database. 9 "6. Steve Denham to be advised the number of 10 residual events and will discuss with Mik Peach. 11 "7. Residual events to be reviewed. 12 "8. Penny (or cover) will check ARQ data retained 13 in the audit room or retrieve message stores, as 14 required. 15 "9. Pete to update security incident register." 16 So it appears that an agreed action for you, 17 although you are not on the recipient list here, was for 18 you to run the event check through the database. Can 19 you help with what the event check was? 20 A. Yes, well, I can remember this moderately well. It was 21 all automated later on but, before I joined the Audit 22 Team, as I said, they brought all the events back from 23 the audit system for all counters and they stored them 24 all in a database, so that, for any given date -- date 25 range and I assume FAD code too -- I can't remember it 141 1 specifically, I imagine FAD code too -- you can get all 2 the post office counter events output in a spreadsheet, 3 which could be supplied. 4 Q. Ms Thomas' email was then forwarded to you by Steve 5 Evans, scrolling up the page, please, with a request to 6 discuss: 7 "Gerald 8 "We will need to discuss this (below) in the AM. 9 "Steve." 10 Do you recall discussing the task case you had been 11 allocated with Mr Evans. 12 A. I remember the task I don't remember specifically 13 discussing it but, if it says in the email this was 14 going to happen, I imagine it did. I certainly remember 15 the database of all the events. I remember that quite 16 clearly. 17 Q. Could we have on screen, please, FUJ00155421. About 18 a third of the way down the page, we have an email from 19 Penny Thomas to Dave Posnett from the Post Office. It 20 is dated 4 February 2009. The subject line is "Security 21 Incident". Ms Thomas says this: 22 "We are pleased to advise that our analysis of data 23 covering 1 May '07 to 30 November '08 has been 24 completed. 25 "The event logs have been checked for all data 142 1 provided to POL as a result of the 195 ARQs which fall 2 within the time frame. A total of 27 instances of 3 concern were identified. All instances have been fully 4 analysed and we can confirm that the locking was caused 5 by contention between the EoD process and a Riposte 6 checkpoint being written. No transactions or balancing 7 activities carried out at the branches were affected." 8 There is reference here to the 195 ARQs which fell 9 within the time frame. There was a reference in Penny 10 Thomas' email of the 8 January 2009 to there being 195 11 outlets within the time frame. Would you agree, 12 therefore, that this email seems to be referring to the 13 same issue? 14 A. I would say almost certainly but, I mean, I couldn't be 15 100 per cent certain, I suppose. But I would say very 16 likely. 17 Q. Do you recall being made aware of the outcome of the 18 checks that were done on the data provided to the Post 19 Office? 20 A. No, it was all -- unless it was copied to me in an email 21 and I didn't read it or something, it was all -- I was 22 aware of the checking of events, but the reason it was 23 done was, at the time, not something I was aware of, 24 though now I can see the reason. But, at the time, 25 I would -- I don't think I was aware, actually, no. 143 1 Q. Your task of running the event check through the 2 database, was that to return results which were sent on 3 to others for analysis? 4 A. That's correct, yes. That's correct. 5 Q. You offer some reflections on the CABSProcess issue at 6 paragraph 33 of your first statement. Could we have 7 that on screen, please. It is page 16 of WITN09870100. 8 You say here at paragraph 33: 9 "The CABSProcess issue highlighted a problem that 10 could easily be caused by another system process at any 11 time of day. In retrospect, error handling should have 12 been tightened generally. For example, when I wrote the 13 software to migrate from Legacy Horizon to HNG-X, I kept 14 this in mind. The postmaster pressed the migration 15 button which appeared on migration day and if anything 16 went wrong the postmaster got a message displayed saying 17 something to the effect of: 'An error has occurred 18 please contact the Helpdesk'. The program then stopped 19 further processing and detailed evidence was recorded 20 that would enable the Helpdesk to identify the issue 21 (possibly after escalating the issue to me). In my 22 opinion, this sort of error handling is the safest. 23 When something goes wrong everyone knows about it 24 immediately and nothing is committed -- in this case, 25 the post office branch would not be migrated and needed 144 1 to continue using Legacy Horizon a bit longer." 2 Does it follow from what you say in this paragraph 3 that, in retrospect, error handling should have been 4 tightened generally, that although there was a fix done 5 following the CABSProcess issue, as far as you were 6 aware, there was not a wider change to coding to prevent 7 silent failures in the system? 8 A. Not in a -- not in a comprehensive manner. I think 9 little improvements were done all the time but I think, 10 ideally, just as when I designed this migration 11 software, before they even started, they should consider 12 the possibility of some system code failing. What do we 13 do if that happens? And when you're choosing a cash 14 account, the obvious thing to do is just display 15 a message to the postmaster that "An error has happened, 16 please contact the Helpdesk". Just as in the migration 17 software, similar thing. Anything goes wrong, just log 18 as much information as possible, and just say to the 19 postmaster clearly and precisely "Please contact the 20 Helpdesk". Don't just sort of roll over silently as 21 though he thinks it's all -- everything is fine when it 22 isn't. 23 That, in my opinion, no -- I mean, retro -- 24 hindsight is a wonderful thing, isn't it, but, in my 25 opinion, that's the way error handling should have been 145 1 done. 2 Q. Should the Chair take it from this paragraph that you 3 consider the CABSProcess issue was a missed opportunity 4 to address deficient coding practices which led to 5 silent failures? 6 A. No, well, I mean, I think it was two -- we're just about 7 to replace Horizon with HNG-X so the better thing to do 8 would be to make sure the HNG-X software learned the 9 lesson, I think. It would just have been too expensive 10 to do a thorough job at that stage. 11 Q. There was, in fact, another issue with which you had 12 involvement in January 2008, in addition to the 13 CABSProcess issue which caused you to comment on the 14 adequacy of the error handling process, and that's one 15 that's addressed at paragraph 38(a) of your first 16 statement. 17 Could we have on screen, please, FUJ00155224. This 18 is a clone of another PEAK and this cloned call contains 19 some comments from you following the report of a stock 20 unit rollover issue which was being experienced by 21 a user in branch. 22 Could we go to page 6 of this document, please. The 23 second entry on this page is made by you and is dated 24 15 January 2008. Starting on the second line of your 25 entry, you say this: 146 1 "The problem was in fact already flagged. A message 2 in the audit log pinpointed the precise message that 3 caused the problem. 4 "The error handling of balancing is deficient in 5 some ways. In most cases an error is just logged and 6 the code blunders on regardless leaving the system 7 locked. What should happen is that the error should be 8 logged, the process cleanly aborted, an error message 9 displayed to the user and the system left so that he can 10 do something else. I hope the HNG-X version is much 11 better. I am not sure it is worthwhile spending time 12 improving the EPOSS version which is shortly to be 13 replaced; it would be better spending the same effort 14 making the HNG-X version better. I had already 15 requested that this be advised to the HNG-X team in 16 PC0152376." 17 So you were, once again, flagging the error handling 18 problem as you saw it? 19 A. That's right, yes. 20 Q. As far as you were aware, is it right that no material 21 changes were made to this wider problem relating to 22 error handling at the time? 23 A. Well, it would have just been uneconomic, it was too 24 late but we were always doing little improvements, 25 though, but it was just -- it would have just been -- 147 1 you see, once the system is rolled out and you start -- 2 is in maintenance roll and developers, like I, are just 3 maintaining it, always, when you do a fix, you really 4 want the minimum code change to effect -- to solve the 5 problem because it reduces the amount of regression 6 testing needed for the release. 7 To comprehensively rewrite the error handling would 8 just be a massive job. That would be a massive 9 regression testing exercise and so would be extremely 10 expensive and, since it was just being rewritten anyway, 11 it seemed particularly pointless. 12 Q. You say in your statement that you bore in mind the need 13 for good error handling processes, when you wrote the 14 software to migrate from Legacy Horizon to Horizon 15 Online and you've just discussed the issues there would 16 have been by, on a shorter term basis, making changes. 17 Can you help with whether there were any other steps 18 taken by you or anyone else within Fujitsu to ensure 19 that good error handling processes were introduced 20 across the board, either at the time of the migration to 21 Horizon Online, or later? 22 A. Well, I can't -- I passed on my comments to the HNG-X 23 team, I hope they got passed on. I can't say, though, 24 I was not really involved in that software. Certainly, 25 the migration software I wrote, very much took that into 148 1 account and I wrote it with the very comprehensive error 2 handling in the first instance and, indeed, every 3 counter was migrated from Horizon to HNG-X with not very 4 many issues, really. 5 MS PRICE: Sir, I wonder if that might be a convenient 6 moment for a short afternoon break. I think you're on 7 mute, sir. 8 SIR WYN WILLIAMS: Does short mean less than 15 minutes? 9 MS PRICE: Yes, please. Ten minutes, sir, if we could. 10 SIR WYN WILLIAMS: Okay. Ten minutes. So when do we start? 11 MS PRICE: That takes us to 3.35. 12 SIR WYN WILLIAMS: Right, thank you. 13 MS PRICE: Thank you, sir. 14 (3.23 pm) 15 (A short break) 16 (3.35 pm) 17 MS PRICE: Hello, sir, can you see and hear us? 18 SIR WYN WILLIAMS: Yes, thank you. 19 MS PRICE: Mr Barnes, turning, please, to events after you 20 joined the Audit Team in 2009. You say in your 21 statement at paragraph 13 that when you joined the Audit 22 Team, Fujitsu was changing from using Escher's Riposte 23 software to Fujitsu's own bespoke software, HNG-X. Can 24 you explain, please, what that meant for the software 25 which was used to perform ARQs for the Post Office? 149 1 A. Right, well, it's -- because we're no longer using 2 Escher's Riposte system -- well, there are two things. 3 First of all, the audit software took -- used the Escher 4 software to produce its spreadsheets of results, which 5 I only discovered this through reading, really. 6 I wasn't there at the time. But it used the Escher 7 software to produce its spreadsheets and results, so 8 that was one thing. So, therefore, just to save the 9 licence fee that we paid to Escher, we wanted to get rid 10 of that component. 11 But, on top of that, in addition, the Audit Team had 12 to cope with the new transactions which were going to be 13 written by the new HNG-X software, which was a Fujitsu 14 rewrite of the -- everything that was done by Riposte 15 before was going to be rewritten by Fujitsu. 16 So the audit software had to cope with this new 17 format transactions too and so a team that mainly 18 completed their work before I joined the Audit Team 19 wrote a component called the Query Manager Service, 20 whose purpose was to produce the spreadsheets very 21 similar to that which was produced by the old audit 22 system for Riposte and, in addition, at the same time, 23 enable it to produce those spreadsheets for the new 24 HNG-X software. 25 Q. Is it right that it was -- is it XQilla -- 150 1 A. XQilla. 2 Q. -- which was used after you joined the Audit Team to run 3 audit queries? 4 A. That's correct, yes. That's right. 5 Q. Is it right that the Audit Team still uses XQilla to run 6 audit queries today? 7 A. That's correct, yes. 8 Q. Does it follow from the fact that you only joined the 9 Audit Team in around 2009 that you would not have been 10 familiar with the high-level design documents relating 11 to Legacy Horizon, covering the design and requirements 12 of the audit harvester created in the early 2000s? 13 A. No, I wouldn't have been -- I wasn't very aware of that. 14 I knew a little bit about it but I wasn't aware of the 15 details. 16 Q. You deal with the audit query process for Horizon Online 17 at paragraph 16 of your first statement. Could we have 18 that on screen, please. It is page 6 of WITN09870100. 19 At paragraph 16, you say: 20 "In relation to HNG-X, the process of generating 21 a spreadsheet of transactions (similar to the ARQ 22 spreadsheet) is as follows: 23 "a. Files to be audited are placed on many 'shares' 24 across the estate. A share is a folder of a compare 25 that is accessible by another computer. 151 1 "b. 'Gatherers' on the audit server bring the files 2 into the audit server, where they are stored on 3 a special long-term storage device (known as an audit 4 archive -- Centera to begin with, which was later 5 replaced by Eternus) and indexed using a Structured 6 Query Language ('SQL') database on the audit server. 7 A checksum of the file is also stored too (a checksum is 8 effectively a unique numerical identifier that is 9 allocated to a file). 10 "c. A special tool on audit workstations can then 11 be used to display stored files and retrieve them. As 12 these stored files are retrieved, their checksum is 13 checked. Some of the stored files are files of 14 transactions and extra software is available to generate 15 spreadsheets of transactions." 16 Is it right that you were not personally involved in 17 responding to ARQs that were submitted by the Post 18 Office to Fujitsu in relation to investigations, court 19 proceedings or disciplinary proceedings? 20 A. That's correct. I simply was part of the team that 21 supported the software they used. 22 Q. You do address in your statements, however, your 23 involvement in a number of issues which could affect the 24 accuracy of ARQ data, and one of these is the duplicate 25 transactions issue which arose in 2010. You've 152 1 addressed this issue at paragraphs 34 to 37 of your 2 first statement, and at paragraph 34, you describe the 3 result of the issue which arose to have been that 4 multiple instances of one transaction could appear on 5 a spreadsheet generated as part of the ARQ process and 6 it would not be clear that they were the same 7 transaction. Is that an accurate summary? 8 A. Exactly. That was the problem, yes. 9 Q. Could we have on screen, please, FUJ00172183. This is 10 a PEAK with reference PC0200468. The summary reads: 11 "Horizon Audit Spreadsheet Producing Duplicate 12 Transactions." 13 There is an impact statement, dated 23 June 2010, 14 which says this: 15 "From Penny -- In a nutshell the HNG-X application 16 is not removing duplicate transactions (which may have 17 been recorded twice on the audit server) and they are 18 appearing in the ARQ returns. For the old Horizon 19 application Riposte automatically removed duplicate 20 entries. An initial analysis shows that one-third of 21 all ARQ returns (since the new application has been in 22 play) have duplicated transactions." 23 Going then to the entries in the log themselves, the 24 second entry is dated 21 June 2010 and is made by Penny 25 Thomas, and she says: 153 1 "While performing an audit retrieval for branch 2 072128 duplicate transactions have been found on 3 June 3 '09. Initial analysis shows that duplicate records are 4 held in 2 different audited TMS files." 5 Then scrolling down to the final entry on this page, 6 please, 22 June 2010, this is an entry made by you, and 7 we have this: 8 "The processing is done by QueryDLL.dll. The way it 9 works is that it processes all the results in a given 10 file, building up an internal table of transaction 11 sequences for that file. Then at the very end of 12 processing the file it dumps the internal table to the 13 RFIQueryFileSequence table. It does not cross-check the 14 transactions in one file against those in another file." 15 You say that: 16 "Two solutions are possible. 17 "The 'easy solution'. 18 "As each transaction is processed a check is made 19 with the RFIQueryFileSequence table and if it is already 20 there the transaction is ignored writing a warning to 21 the query log. The problem with this solution is that 22 a query needs to be made to the database for every 23 transaction. 24 "The 'more difficult solution'. 25 "The internal table which at the moment is built up 154 1 on a per file basis is changed to being built up on 2 a per query basis. The check for duplicate transactions 3 is then done within the internal table. This is a much 4 more thorough approach but will take much more work." 5 Then there is a further entry from you, also dated 6 22 June, which outlines the detail of the fix for this 7 problem, and about halfway down the page we have "Impact 8 on user", and "Impact on User" says: 9 "Occasionally duplicate transactions are listed in 10 the spreadsheets produced and presented to court for 11 prosecution cases. These can give the defence team 12 grounds to question the evidence." 13 Then further down, in response to the question, 14 "Have relevant KELs been created or updated?", you say: 15 "No KELs have been created for this since we intend 16 to fully resolve the issue shortly." 17 If we scroll down, there are risks that are outlined 18 of releasing and not releasing a fix, and here you say: 19 "If we do not fix this problem our spreadsheets 20 presented in court are liable to be brought into doubt 21 if duplicate transactions are spotted." 22 Going over the page, please, the entry of 23 June 23 2010, from Penny Thomas: 24 "Initial analysis of all ARQ returns since the HNG-X 25 application has been implemented identifies 155 1 approximately one third (of all returns) have duplicate 2 entries. This is now extremely urgent." 3 Scrolling down, please, towards the bottom of the 4 page, there is an entry on 7 July 2010. Right at the 5 bottom of the page, that last entry, it says: 6 "Fix Released to PIT." 7 Can you just help with what that means? 8 A. Yes, that's the team which generated the actual thing 9 that was automatically delivered to the various 10 platforms. In this case, it would have been the audit 11 server. 12 Q. Then over the page -- 13 A. Post Office Integration Team, possibly. I can't -- I'm 14 just trying to think what -- maybe Post Office 15 Integrate -- I'm guessing -- or Pathway Integration 16 Team. Something like that. 17 Q. Then over the page, please, to the second entry of 18 7 July 2010. It says here: 19 "PEAK has been test installed in Integration." 20 What does this mean, please? 21 A. Well, integration, that's the PIT team, that they 22 actually produce -- so development produce V baselines, 23 with the fix in, and that goes to the integration team 24 who produce D baselines, which are ready for automatic 25 deployment on the various platforms which, in this case, 156 1 would be the audit server. 2 Q. There are then a series of entries which follow before 3 the final entry on this page, dated 1 September 2010, 4 made by Penny Thomas, which notes: 5 "Fix successfully deployed, closing call." 6 So it seems from this PEAK that, although the issue 7 was being raised in June 2010, it was not the subject of 8 a fix until 1 September 2010; is that right, from what 9 we can see on this log? 10 A. Well, that's certainly when Penny could see it was 11 successfully deployed. Well, roughly speaking, yes. 12 It's the -- you've got 30 July, John Rogers tested 13 successfully, completed and documented in LST. Yeah, it 14 takes -- after it gets tested in LST, it doesn't usually 15 take long before it's deployed to live, so from 30 July 16 to 1 September, um, sounds a bit of a long time, but 17 anyway that sort of thing, yes. That sort of thing. 18 Q. Can we have on screen please FUJ00171848. The reference 19 on this PEAK is PC0205805. The summary is: 20 "Audit -- Duplicate Message sequences and not 21 recorded by Fast ARQ retrieval." 22 The call was opened on 27 October 2010, if we can 23 scroll down a little to see that. Before we turn to the 24 detail of the PEAK, can you explain, please, the 25 difference between fast and slow ARQs? 157 1 A. Right, well, the slow ARQ was the original method, which 2 was a sort of more labour intensive system for the 3 operator but more flexible, in which, first of all, they 4 had to supply the FAD, the date range and the files they 5 wanted to get back, and then they went to another screen 6 to supply the date range they wanted to filter the FAD 7 for, and then they went to another screen to supply the 8 query they wanted to run on the FAD. 9 So it was all quite doable but someone called Steve 10 Meek, whose name has come up before, automated this 11 process, so you had one screen, which you have the FAD, 12 the date range and, also, an optional number of extra 13 days for files gathered late and you just set it going 14 with one form. So it was just quicker for the operator, 15 the fast ARQ was quicker for the operator. 16 Q. The issue addressed in this PEAK arose in the context of 17 fast ARQs; is that right? 18 A. That's correct, yes. Well, that's what it says. 19 I mean, that's what it says. 20 Q. The "Impact Statement", dated 5 November 2010 appears to 21 have been entered by you and says this: 22 "The Fast ARQ interface does not provide the user 23 with any indication of duplicate records/messages. 24 "This omission means that we are unaware of the 25 presence of duplicate transactions. In the event that 158 1 duplicates are retrieved and returned to POL without our 2 knowledge the integrity of the data provided comes into 3 question. The customer and indeed the defence and the 4 court would assume that the duplicates were bona fide 5 transactions and this would be incorrect. There are 6 a number of high profile court cases in the pipeline and 7 it is imperative that we provide sound, accurate 8 records." 9 Looking then, please, to the entry at the bottom of 10 this page, dated 1 November 2010, again this is an entry 11 made by you. It says: 12 "Andy and I have looked at this. We think the 13 method most compatible with existing behaviour is as 14 follows -- 15 "Check for duplicates for HNG-X in a similar method 16 to how duplicates are checked for in Horizon." 17 Do you mean Legacy Horizon? 18 A. That's right, that's correct, yes. 19 Q. "For Horizon they are legitimately logged in the audit 20 log and then are ignored (because it is just that 21 identical medicines as stored by mistake in more than 22 one transaction file). For HNG-X, in the Fast ARQ case, 23 their detection will cause them to be logged in the 24 QueryLog and a count kept of how many there are; they 25 will not be ignored." 159 1 Then you go on to detail a proposed fix. 2 You made a further entry on 5 November 2010, which 3 reads: 4 "I have built a prototype QueryDLL.dll which solves 5 this problem. Now if duplicate HNG-X messages are 6 detected the Fast ARQ fails at the client with the 7 message 'filtering failed' displayed at the bottom of 8 its form and on the server in the QueryLog there are 9 detailed messages describing the duplicates found." 10 Then there is a further, more detailed entry from 11 you, also dated 5 November 2010, where you provide 12 a technical summary, and you say: 13 "HNG-X can rarely produce transactions with 14 duplicate Journal Sequence Numbers. At the moment, when 15 running a FAS ARQ on the audit server, these duplicates 16 are not noticed. This means that the evidence presented 17 by the Prosecution Team may show duplicate transactions 18 without it being noticed; the Defence Team may spot this 19 and call into question the integrity of our data." 20 Scrolling down, please, to the bottom of the page 21 "Impact on User": 22 "HNG-X transactions with duplicate JSNs may not be 23 noticed. This will call into question the reliability 24 of evidence present by the prosecution team." 25 Going to the top of the next page, please, on "Have 160 1 relevant KELs been created or updated?": 2 "It was not felt that a KEL was required because 3 there are only two people in the prosecution team and 4 they are both fully aware of the problem." 5 Who were the two people in the prosecution team; can 6 you recall? 7 A. Well, Penny Thomas would have been one. 8 Q. Do you recall Andy Dunks? 9 A. Well, he certainly was there at one -- well, still is, 10 still is -- might have been Andy I was referring to 11 I can't -- quite likely. I would say quite likely but 12 I couldn't say with certainty. The only one can say 13 with certainty is Penny Thomas. 14 Q. Whose decision was it whether a Known Error Log was 15 created? 16 A. Well, I think that's -- if I'm writing this, then it's 17 me, really, I suppose. You've either got to produce 18 something in the KEL or you've got to give 19 an explanation of why you're not producing something in 20 the KEL. Well, my explanation is quite simple: there's 21 only two people and they're well aware of it anyway, so 22 there's no point having a KEL entry because they know 23 about the issue. 24 Q. This PEAK was opened around two months after the last 25 PEAK we looked at was closed, following a fix on 161 1 1 September 2010. Is the issue discussed in this PEAK 2 the same as that discussed in the previous PEAK? 3 A. Yes, that's right. That's right. It would have been -- 4 the same fix as applied before, I think, would have been 5 present here. So you get -- whenever you do the ARQ, 6 you get -- the query handler log is always generated. 7 That would have listed the duplicates here anyway. 8 I can't quite remember what we're fixing here. Not too 9 sure exactly what we were doing. But -- 10 Q. Can you help -- apologies. I interrupted. 11 A. Well, yeah, I'm not -- what -- I'm not sure that Andy 12 and -- I'm not sure exactly what change would have been 13 going on here. Does it say somewhere exactly what we 14 were going to do to address this issue? 15 Q. Well, the -- going a little further back up the page, if 16 we can just stop there, just casting your eye down 17 there, does that help you at all? It says, for example: 18 "Does the fix require any manual deployment 19 baselines? 20 "The fix does not require any manual installation; 21 it would just be a replacement file ... 22 "The coding of the fix is complete, however further 23 regression tests still need to be run." 24 A. Okay, well, I can't remember exact -- all I know is what 25 happens right now. The intermediate steps of how we got 162 1 there, I'm not too sure of. 2 Q. Putting it another way, why was there still an issue if 3 a fix had been implemented on 1 September? 4 A. I'm not too sure, to be honest. I would have thought on 5 the first fix we'd logged all the duplicates in the 6 query handler log. That would have been the case here, 7 whether we're doing something more. Maybe, what it 8 might have been, is we simply -- ooh. Yes, it could be 9 that we actually got the Fast ARQ to actually say: 10 right, duplicates have occurred we're stopping. It 11 could just have been that the Fast ARQ just failed, and 12 you'd get just some message saying "Look in the query 13 handler log" and then you would see the duplicates 14 listed. 15 It might have been that I can't, from memory, 16 remember exactly what the fix was. It might have been 17 that. So maybe it just stopped running and refused to 18 produce anything. 19 Q. Scrolling down, please, and over the page, looking for 20 an entry on 24 November 2010. 21 A. Oh, Andrew has written something. 22 Q. That entry there, 24 November 2010, Andrew Mansfield: 23 "Sarah Selwyn has requested an audit maintenance 24 release prior to the next DC_AUDIT planned release due 25 to go live on 14/05/2011. 163 1 "Five PEAKs are requested for this maintenance 2 release [and they're listed]. 3 "This is an edited version of the text of Sarah's 4 original email to Sheila Bamber: 5 "We would likely get these PEAKs targeted ASAP since 6 these are impacting SSC and the Litigation Support Group 7 in their support of the Post Office litigations. There 8 is a risk that these teams will not be able to fulfil 9 their OLTs to the Post Office as defined in 10 SVM/SDM/SD/0017 ..." 11 In terms of when the issue was fixed, you refer to 12 another PEAK at paragraph 37 of your first statement, 13 which you say suggests that the issue with duplicate 14 transactions was fixed in or around November 2010. 15 Could we have that on screen, please. It's FUJ00171892. 16 The reference for this PEAK is PC0205353. The summary 17 reads: 18 "LST -- Audit -- Duplicate message sequences are not 19 reported if they are identical." 20 This not a PEAK log which contains entries made by 21 you but, since you refer to it in your statement, can we 22 take it that you have reviewed it? 23 A. Yes, yes, yes, yes. 24 Q. The "Impact Statement" here says: 25 "It is important that any duplicate messages in the 164 1 retrieved audit data are highlighted to the user. 2 "Duplicates are not being highlighted when two 3 message sequences have the same start and end message 4 sequence numbers." 5 So it gives somebody examples: sequences X to Y 6 would not be reported as duplicates; sequences X to Y 7 would report a duplicate: 8 "This is a very serious issue. We experienced the 9 presence of duplicate Horizon transactions which were 10 not removed when the HNG-X application was introduced. 11 POL did not accept a manual workaround and the ARQ 12 service basically stopped for almost 2 months. 13 "The issue contained in this PEAK came to light on 14 21 October and I have instigated the creation of a macro 15 which will identify if duplicated transactions are 16 contained within a spreadsheet. We will need to 17 generate an additional spreadsheet containing the JVN 18 and check for duplicates by using the macro. This will 19 increase our workload by 15-20 minutes for each ARQ 20 containing HNG-X transaction records. 21 "The real problem will arise if we do identify 22 duplicate transactions because POL is not likely to 23 accept a workaround for transaction records used for 24 Litigation Support." 25 That statement is dated 25 October 2010. Is this 165 1 duplicates issue as the Fast ARQ -- 2 A. No, no, it's a much more -- it looks like a much more 3 specific issue. It's -- "When two message sequences 4 have the same start and end message sequence numbers", 5 so it looks like some very specific issue that Andy 6 Mansfield fixed. 7 Q. If we could go to page 5 of this document, please, top 8 of the page, an entry on 25 November 2010 reads: 9 "Cleared in release 3.13 (Audit System) and tested 10 in LST under release notes HRU7206 and HRU7239. 11 "Closing call." 12 Was this the entry which led you to believe that the 13 duplicate transactions issue had been resolved in or 14 around November 2010? This is the document reference -- 15 A. Yes, I think so, I mean, I get a bit confused with the 16 exact timescale but what happens right now is that 17 spreadsheets we send to the -- and that has been for 18 a long time -- is that the spreadsheets we send always 19 have in a summary sheet details of all the duplicates 20 and gaps. So each spreadsheet we submit has all that 21 information in the spreadsheet. That's the present and 22 been that way for a while, but getting there has been 23 a slightly not so -- we haven't got there in one go, as 24 it were. That is the way it's been for a long while. 25 Q. The last entry we looked at in this was on page 4. 166 1 Apologies, if we can have back on screen, please, 2 FUJ00171848. Looking at page 4, please. Going back to 3 Andrew Mansfield's entry dated 24 November 2010, that 4 was the last entry that we looked at. 5 Going, please, to the top of the next page to 6 page 5. There's an entry from you, dated 3 December 7 2010, saying: 8 "A fix will now be prepared and tested. It will 9 then be stored in VSS-InfDom. It will be handed over on 10 24 December." 11 There is then an entry on 14 December from you and 12 you say: 13 "It has now been decided that the detection of 14 duplicate HNG-X messages will not terminate the FAST 15 ARCs." 16 Is that supposed to be ARQs? 17 A. That's right, yes. 18 Q. "Duplicates will be logged by QueryDLL at the server 19 initially in the QueryHandler.log and eventually in the 20 close log both for Horizon and HNG-X transactions. 21 Duplicate HNG-X transactions will also be logged by the 22 client in its spreadsheet but duplicate Horizon 23 transactions will be eliminated at the server silently 24 since they are known always to be benign." 25 Then on 29 December the entry from you, we have: 167 1 "Fixed by version 4.1.0.1 of NWB_Legato_Recover.exe 2 and version 4.0.0.4 of QueryDLL.dll handed over in 3 AUDIT_EXTRACT_SVR [and the reference]." 4 Going to the bottom of the page, please, on 5 19 January: 6 "Tested in LST as part of Audit Release 3.24. 7 "Duplicate message sequences are now recorded in the 8 Query Handler and Closure (RFI) log files, for both Slow 9 and Fast ARQs." 10 There are some further entries over the page and 11 a final entry on 27 April 2011, which reads -- there are 12 two entries here, John Budworth first of all: 13 "Applied to live 03/04/2011 as part of Audit Release 14 03.24." 15 Then we have "CALL ... closed" entry by Penny Thomas 16 on 27 April 2011. 17 Having looked at this document, does it remain your 18 understanding that the duplicate transactions issue was 19 fixed in or around November 2010 or do you think that 20 may have been later? 21 A. It looks like it might have been later, actually, yes, 22 from this. A complete fix, yes, it looks like it might 23 have been later. 24 Q. At paragraph 36(b) of your statement you address another 25 problem which arose in relation to fast ARQs relating 168 1 not to duplicate transactions but to missing 2 transactions. Could we have on screen, please, 3 FUJ00171894. The reference for this PEAK is PC0207787, 4 the summary is: 5 "Audit -- Transaction Gap info overwritten in 6 Summary worksheet." 7 The "Impact Statement" written on 18 January 2011 8 says: 9 "The problem will only occur in exceptional 10 circumstances but should be fixed in case the 11 exceptional circumstance happens. 12 "If it does occur, transaction gap information is 13 overwritten in the results spreadsheet and we would not 14 be able to send the ARQ to POL. We would probably 15 attempt to resolve the cause of the gaps or duplicates 16 before sending the output to POL in any case, but the 17 problem really ought to be fixed." 18 Recognising that this is not a PEAK log into which 19 you made entries but one you commented on in your 20 statement, is this problem a distinct one from the 21 duplicates problem we've been looking at -- 22 A. Um -- 23 Q. -- or part of the same problem? 24 A. No, this is -- this appears to be about gaps. So you've 25 got duplicates and you've got gaps. So all the messages 169 1 written have a message number and you can have 2 a duplicate, but also, very rarely -- and it is very, 3 very rare, actually -- you can have a gap, ie no message 4 at all. This is referring to gaps, which -- duplicates 5 are quite common, particularly for -- well, actually 6 duplicates were quite common for Horizon but not for 7 HNG-X, really, I don't think. 8 Gaps weren't common for anything but you've got to 9 check for them and this seems to be saying that there 10 was some problem with reporting gaps. 11 Q. Going, please, to page 4 of the document, there's 12 an entry of 20 June 2011 there: 13 "PEAK has been test installed in integration, 14 routing back to source." 15 Then there are three entries on 30 June 2011, one 16 from Mark Ascott saying, "Successfully tested by LST", 17 and one final one closing the PEAK. 18 Can you help with whether there was a further fix in 19 relation to this issue, in addition to the fix 20 implemented for duplicate transactions, or is this part 21 of the same thing? 22 A. Well, I assume this is a separate issue. I assume this 23 is a separate one. 24 Q. In your first statement you addressed a number of other 25 PEAKs which you consider may have had an impact on the 170 1 audit log. I don't propose to take you to all of those, 2 but one of these relates to an issue arising in April 3 2013. Could we have on screen, please, FUJ00226106. 4 About two-thirds of the way down the page, there's an 5 email from you to CSPOA Security. Can you just explain 6 please which team that was? 7 A. That's Cyber Security Post Office Account, which is -- 8 which is the same as the -- well, it's what I sometimes 9 call the prosecution team, sometimes it's -- I suppose 10 it's not -- I suppose -- it's part of the same thing, 11 yes. It's the same -- Penny, et cetera, at the time. 12 Q. It is copied to Rajbinder Bains and Andy Dunks, among 13 others. It is dated 15 April -- apologies, if we can go 14 down, please, that email below, 15 April, an email from 15 you. The subject line is "Possibility of missing 16 transactions", and you say: 17 "Hi, 18 "A serious flaw has recently been spotted in the 19 audit code. It was introduced in the fix to PC0187097 20 quite some time ago (but post-HNG-X). There is a small 21 possibility of missing transactions on generated 22 spreadsheets if the query handling was run during the 23 evening Query Manager shutdown. Please raise a priority 24 PEAK on this issue and send it to Audit-Dev." 25 Mr Dunks replied to you the next day, a bit further 171 1 up the page, please, saying this: 2 "Gerald, 3 "Can you confirm that we're talking about as far 4 back as September 2009? 5 "Are you able to pop down and explain and show us 6 what we are to look for, as we will need to put together 7 some time scales to complete this task." 8 Then above, you reply and say: 9 "I will come down in a few minutes." 10 Just to be clear, the issue you were flagging 11 appears to have the potential to lead to transactions 12 missing from audit data provided to the Post Office by 13 Fujitsu; is that right? 14 A. Yes, that's correct. Though, in fact, it would be quite 15 likely -- it might be noticed by our gap checks because, 16 if something was getting missed, unless it was at the 17 very beginning, at the very end of the range, it would 18 be noticed by the gap check but, yes, potentially. 19 Potentially. 20 Q. Was Andy Dunks right when he was asking if this could go 21 back as far as September 2009? Do you know how far it 22 could have gone back? 23 A. Well, not -- not offhand. You'd need to look at the -- 24 when this PEAK I cross-referred to got fixed, I suppose. 25 But, well, if he said that, I suppose possibly -- 172 1 possibly, although without going into detail, I couldn't 2 say. 3 Q. Could we have on screen, please, FUJ00173057. This is 4 PEAK reference PC0225071. The summary is "Possibility 5 of missing transactions on ARQ audit spreadsheets", and 6 the "Impact Statement", dated 12 June 2018 is entered by 7 you. It says this: 8 "There is a loophole in the code of QueryDLL.dll 9 whereby if it is running during the evening service 10 shutdown the resulting prosecution spreadsheets produced 11 later may have missing transactions. 12 "There is a tiny possibility that errors in the 13 QueryManager service may not be reported meaning that 14 invalid prosecution spreadsheets may be produced. 15 "There is a possibility of errors being generated 16 when audit queries are being run and the QueryManager 17 service is shutdown and restarted. This wastes the time 18 of the prosecution service and makes them rerun queries. 19 This makes achieving SLAs more difficult." 20 You appear from the log entries in this PEAK to have 21 been involved in investigating and finding a fix to this 22 problem. Looking on page 2, please, to an entry of 23 16 April 2013, there is a reference -- going back up, 24 please -- there's a reference to a meeting held the day 25 before with Adam Spurgeon, Alan Holmes and Steve 173 1 Goddard. Can you help with which teams these 2 individuals were in? 3 A. Alan Holmes was the Manager of the Audit Team at the 4 time -- not the Manager, was the Designer for the Audit 5 Team at the time. Steve Goddard and Adam Spurgeon were 6 Managers. 7 Q. Scrolling -- if you can't assist any further -- 8 A. Well, I thought I'd answered the question. 9 Q. I'm sorry, I thought you were continuing. 10 A. No, that was it. Yes, there was -- so Alan Holmes was 11 the Designer, and the other two were Managers. 12 Q. Scrolling -- 13 SIR WYN WILLIAMS: Before we get into this document, can 14 I tell you, Ms Price, that I can't go on beyond 4.30 15 today. So, since we're getting reasonably close there, 16 I think we'd better take stock about what's happening. 17 MS PRICE: Yes, sir. The witness is available to attend 18 tomorrow morning to finish his evidence, should that be 19 necessary. I was going to stop after this topic to see, 20 sir, whether you wanted to sit a little later or to 21 continue tomorrow. 22 SIR WYN WILLIAMS: Well, I think we are going to have to 23 continue tomorrow by the sound of it. So just choose 24 a suitable moment between now and 4.30 to round off 25 then, all right? 174 1 MS PRICE: Yes, sir. 2 Scrolling down, please, to the last box on this 3 page, the 16 April entry, you say that the affected 4 platform was the audit server and the technical summary 5 was: 6 "A loophole has been found in QueryDLL.dll whereby 7 if it is running during the evening shutdown of the 8 QueryManager service the prosecution spreadsheets 9 produced later may have missing transactions. 10 "In addition the design ethos at the moment of 11 QueryDLL is that on shutdown a failure state is 12 indicated. This is to be changed to there being a rerun 13 of the query after shutdown which would have prevented 14 this problem in the first place, although there would 15 still have been a problem if a genuine error rather than 16 a shutdown had occurred prior to the faulty code which 17 masked the earlier state. 18 "As well as that and as a precaution the error 19 handling of QueryDLL.dll is going to be looked at and 20 improved." 21 The "Impact on User" is dealt with a little further 22 down, going to the top of the next page: 23 "The prosecution spreadsheets will be more reliable 24 after this fix ..." 25 There is then an entry towards the bottom of this 175 1 page on 24 May 2013 made by you. In this, you deal with 2 completion of initial testing using a debug version and 3 attach your test plan. You say: 4 "Unfortunately 7.22 has been superseded by an 8.01 5 release and so the fix will need merging ... There has 6 been a debate about where exactly this shall be 7 released. 8 "Whilst investigating the original problem the 9 following problems are fixed in QueryDLL.dll. 10 "The original major problem that transactions would 11 go missing silently from spreadsheets if an evening 12 QueryManager shutdown occurred at a particular point." 13 You go on to explain the other aspects of that. 14 Over the page, we see the fourth entry down, dated 15 12 June 2013, is made by you: 16 "Andy Dunks has stated that he is prepared to only 17 run audit queries in the day to prevent the possibility 18 of audit transactions being missed from spreadsheets due 19 to a bug in the code that handles the overnight shutdown 20 of the QueryManager service. 21 "I am therefore proposing this PEAK for the 9.28 22 maintenance release." 23 This is two months after the issue first arose. Was 24 this the first point at which the process for running 25 audit queries was modified to avoid the risk of 176 1 spreadsheets being affected after the issue was raised 2 in April 2013? 3 A. I suppose so, yes, so that is right. Though Andy had 4 checked all -- so you could tell whether there had been 5 an evening shutdown by looking in the QueryManager log, 6 and Andy had checked them all, I believe, and so we had 7 taken checks to make sure everything was okay, as 8 I understand it. But that's right, yes. 9 Q. As far as you can see from this log and as far as you 10 are aware, was Post Office told about this issue, either 11 by this point, or before the fix in November 2014? 12 A. I don't know the answer to that, I'm afraid. I don't 13 know that. 14 Q. The last entry on this page is dated 12 June 2013 and is 15 made by you. Your "Technical Summary" is: 16 "A thorough review of the QueryManager service has 17 been conducted. One major bug has been found which 18 could result in prosecution spreadsheets having missing 19 transactions if the QueryManager service is shutdown and 20 restarted. 21 "In addition, many less serious issues have been 22 found with the QueryManager service. 23 "There is a tiny possibility that if an error occurs 24 it will not be reported. 25 "The evening shutdown can cause queries to fail that 177 1 would otherwise have worked. 2 "These issues are all fixed." 3 Is it right that your finding reported here, albeit 4 you say these issues are now all fixed, was that 5 an error could have occurred and not have been reported? 6 A. That's what it says, so it must be the case. Yes, 7 that's what it's saying. 8 Q. This is the same concern you had expressed in 2007, was 9 it not, around error handling, that the code should be 10 written in a way that prevents silent failures? 11 A. That's right, exactly. But I thought the query 12 handler -- as I say, I didn't write it all myself. It 13 was something that had been written by a team. 14 I thought it was much better a rehandling than what 15 I saw at EPOSS. Though, as with all things, you can 16 always have little gaps, little mistakes, but I thought 17 in general it was better. It had obviously been 18 designed to trap errors from the word go, this service, 19 and they missed little points but it'd basically been 20 designed to trap errors from the word go. 21 Q. Going over the page, please, about a third of the way 22 down, you address the risks of not delivering the fix. 23 Scrolling up, perhaps: 24 "RISKS (of releasing and not releasing proposed 25 fix): 178 1 "If this fix is not delivered, there is the 2 possibility that incorrect prosecution spreadsheets will 3 be produced. 4 "If this fix is not delivered some prosecution 5 spreadsheet production runs will fail if the evening 6 shutdown occurs in the middle of them." 7 Did you recognise, at the time, how significant 8 a problem this might be? 9 A. Yes, well, that's right. Yes, definitely. That's why 10 we took a lot of -- did a lot of checks. That's right. 11 Q. Did you recognise the risk that incorrect data might be 12 presented in support of Post Office prosecutions? 13 A. Absolutely. But, as I say, a lot of steps were taken to 14 check this hadn't happened. 15 Q. In any of your conversations with Andy Dunks, do you 16 recall him talking about the significance of the problem 17 and the risk that incorrect data might be presented in 18 support of Post Office prosecutions? 19 A. Well, he just agreed that he was going to run the checks 20 that were suggested to him, to -- well, to make sure 21 that his spreadsheets hadn't been reduced in the evening 22 shutdown. Because you can always tell, looking at the 23 query handler log, whether this had happened or not. 24 Q. Do you recall being told about any discussion of this 25 issue with the Post Office? 179 1 A. No, I never -- that never -- that never -- no, 2 I didn't -- conversations about the Post Office never 3 really directly got to me, I don't think. 4 Q. This was an issue which had first come up in June 2013. 5 Can you assist with why it took until November 2014, the 6 date you give in your statement for the issue being 7 fixed, for that to happen? 8 A. Which statement? Which section of the statement? 9 Q. Looking at your first statement -- 10 A. I think I do recall it. 11 SIR WYN WILLIAMS: It's paragraph 38(b) and it's about six 12 lines from the bottom. 13 MS PRICE: Thank you, sir. 14 We can look -- 15 A. 2014 -- oh, right. 16 Q. We can look, if it helps, to the last entry in this 17 PEAK, just going to the last page. 18 A. December 20 -- well, okay. It was finally closed in 19 December 2014. Ah, but, ah, ah, but just a second, 20 19 November 2014, it's got "[Software] Fix Available to 21 Call Logger". 22 Q. If it assists, scrolling up a little, the entries 23 immediately above. 24 A. Yes, so it looks as if it's been released -- the fix was 25 released in November 2014, it's just that Jason has 180 1 closed the PEAK in December 2014. So I think my 2 statement is actually correct. Yes, subsequent -- my 3 statement says that "Subsequently deployed in or around 4 November 2014", which is what the statement by Lorraine 5 Guiblin means, 19 November 2014. 6 Q. So -- 7 A. "[Software] Fix Available to Call Logger." 8 Q. Looking back to that original date that the issue was 9 raised -- I misspoke earlier, it was 16 April 2013 -- 10 can you help with why it took until November 2014 for 11 a fix? 12 A. Oh, right, I'm not sure. I can't remember now. I don't 13 know why it took so long. That seems quite a long time, 14 certainly. 15 SIR WYN WILLIAMS: Right, we'll have to take that up further 16 tomorrow, if necessary. 17 MS PRICE: Sir, that was the last of my questions for today. 18 SIR WYN WILLIAMS: I thought it would have been but I gave 19 you the opportunity to have another go. All right. 20 Well, I'm very sorry, Mr Barnes, that you'll have to 21 return tomorrow but I'm grateful to you that you've made 22 yourself available to come tomorrow. Forget about this 23 case tonight, if you possibly can, don't talk about your 24 evidence and come ready for a much shorter session, 25 I hope and suspect, tomorrow morning. Thank you. 181 1 THE WITNESS: Thank you. 2 (4.30 pm) 3 (The hearing adjourned until 10.00 am the following day) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 182 I N D E X JOHN GRAEME SIMPKINS (affirmed) ...............1 Questioned by MR BEER ........................1 GERALD JAMES BARNES (sworn) .................105 Questioned by MS PRICE ......................105 183