1 Wednesday, 9 November 2022 2 (9.57 am) 3 MR BEER: Good morning, sir, can you see and hear me? 4 SIR WYN WILLIAMS: Yes, thank you. 5 MR BEER: Thank you, may the witness John Simpkins be sworn, 6 please? 7 SIR WYN WILLIAMS: Yes, of course. 8 JOHN SIMPKINS (affirmed) 9 Questioned by MR BEER 10 MR BEER: Good morning, Mr Simpkins. My name is Jason Beer 11 and I ask questions on behalf of the Inquiry. 12 In front of you, there should be a witness statement 13 in your name. 14 A. Yes. 15 Q. Can you turn, please, to the last page of it. It is 16 18 pages in length, dated 4 August 2022, and for the 17 transcript the reference is WITN04110100. 18 On page 18 is that your signature? 19 A. Page 19, yes. 20 Q. It's page 19, is it? You are quite right. Are the 21 contents of it true to the best of your knowledge and 22 belief? 23 A. They are. 24 Q. A copy of that will be uploaded to the Inquiry's 25 website. I'm not going to ask you about every part of 1 1 it, just selected parts. Do you understand? 2 A. I do. 3 Q. Can you tell us your qualifications, please? 4 A. I studied software engineering at University of 5 Birmingham. I am a member of the British Computer 6 Society, Chartered IT Professional and an Incorporated 7 Engineer. 8 Q. You joined Pathway, ICL Pathway Limited, in 1996, 9 July 1996; is that right? 10 A. Yes. 11 Q. As an application developer; is that right? 12 A. Correct. 13 Q. Can we look at paragraph 9 of your witness statement 14 please, which is page 3. Just wait a moment, it will 15 come up on the screen. You say in paragraph 9: 16 "While I was initially taken on as an Application 17 Developer, I only remained in this role for a very short 18 time and did not in fact develop any aspects of the 19 Horizon system myself. During my time as an Application 20 Developer, I worked with Dai Jones to learn the coding 21 language being used at the time." 22 When you were working with Dai Jones, was there any 23 discussion about the quality of the coding language 24 being used at the time? 25 A. No, I was really only training at that time, so I was 2 1 being taught how to interact with the Riposte system. 2 Q. Did Dai Jones ever discuss with you the quality of the 3 Coding on the Escher product known as Riposte? 4 A. Not at that time. 5 Q. I said at any time. 6 A. Yes ... not unless there was a PinICL that was raised on 7 it. 8 Q. I'm sorry? 9 A. Not unless there was any calls raised on the code. 10 Q. Can you recall whether there were? 11 A. There were many calls raised on the code over the years. 12 I don't know whether Dai Jones raised any of those 13 calls. 14 Q. Can you remember any wider discussion with Dai Jones 15 about the quality of the coding on the Escher product 16 Riposte? 17 A. No, I don't. 18 Q. You don't? 19 A. I don't recall any further conversations with Dai Jones. 20 I only worked with him for a very short amount of time. 21 Q. After that initial period of training as an application 22 developer, when you worked with Dai Jones, did you have 23 cause to work with him again? 24 A. No. 25 Q. You say the language was known as Visual Basic: 3 1 "A key role of the development team was to ensure 2 that the Visual Basic coding being used by the time 3 interfaced properly with Escher's software product, 4 Riposte ..." 5 Did it interface properly with Escher's product, 6 Riposte? 7 A. Yes, that was the way it was -- that was the way we 8 interfaced with that product, that was -- 9 Q. I know it was the way. I was asking you did it 10 interface properly? 11 A. Yes. 12 Q. There were no problems with it at all? 13 A. With Riposte or Visual Basic or the interaction? There 14 was ... I don't recall any problems with the Visual 15 Basic and the interaction with the DLs(?) between Visual 16 Basic and the Riposte application. 17 Q. Then over the page, you say: 18 "Access to the Escher source code was only granted 19 to the development team if absolutely necessary." 20 So, to your understanding, it wasn't a question of 21 intellectual property rights preventing any access to 22 the Escher source code; is that right? 23 A. I believe so. I think we had a copy of the source code 24 on the sixth floor in a safe in case it was ever 25 required, but I don't recall it ever being used. 4 1 Q. You say that access was only granted if absolutely 2 necessary. Was it necessary? 3 A. I don't recall it ever being used. 4 Q. But that facility was there? 5 A. That facility was there, yes. 6 Q. It wasn't that you could never have access to it? 7 A. I believe the reason it was there was so that people 8 could have access to it. 9 Q. Yes, thank you. You then moved into the software 10 support centre. Is that called the SSC? 11 A. Correct. It was initially the system support centre and 12 then, I think -- 13 Q. I was about to ask. The term "software support centre" 14 and "SSC", is that used interchangeably sometimes with 15 "system support centre"? 16 A. It is. I believe, originally, it was "system support 17 centre" up until after Mik left and I think it got 18 changed to "software support centre" after that time. 19 Q. You have remained, I think, in the SSC for 26 years now. 20 You are currently a team leader in the SSC? 21 A. That's correct. 22 Q. Before you became a team leader in the SSC -- I think 23 that was in 2010 -- what was your job title? 24 A. Project specialist. 25 Q. Was that the same for the previous 14 years? 5 1 A. Yes, I think everyone had that title, really. 2 Q. What was the role of a project specialist? 3 A. It was to receive tickets or we sometimes had direct 4 email and we would investigate problems on the live 5 system and then potentially reports, as well, to service 6 management. We try and produce workarounds if there was 7 an issue and try and resolve problems that were passed 8 to us, really, on the live estate. 9 Q. Was there a level below a project specialist in the SSC? 10 A. Not by terminology. You had areas of specialism, so 11 there were many products that made up the solution, like 12 the databases and Riposte, the agents and -- and people 13 were specialists in certain areas, but I think they were 14 all called project specialists. You might be working on 15 Tivoli, the rollout database, ACMS, or any of these 16 other areas that you were still, I think, called 17 a project specialist. 18 Q. In, say, 1999/2000 at the rollout stage of Horizon how 19 many people worked in the SSC? 20 A. I think we went up to about 25. 21 Q. At, say, 2010, at rollout stage of Horizon Online, how 22 many people worked in the SSC? 23 A. Probably slightly more. I think Mik was hiring at that 24 time, but yes, I mean -- I think it probably topped out 25 around 30, but maybe around 25 to 30. 6 1 Q. How many team leaders were there in the SSC, say, at the 2 first date that I mentioned, 1999/2000? 3 A. There was only a manager at that point, no team leaders. 4 Q. When were team leaders introduced? 5 A. 2010. 6 Q. So when you became one? 7 A. Yes. So after Mik left -- 2009, I think -- we had Tony 8 Little step in for a while and then Steve then took over 9 in 2010, and he introduced the three team leaders. 10 Q. The Steve you refer to there, is that Steve Parker? 11 A. That's correct. 12 Q. Who did each of the SSC team leaders report to: to him? 13 A. To him. 14 Q. He was the SSC manager; is that right? 15 A. Correct. 16 Q. Do you know to whom he reported? 17 A. Not entirely -- Steve Muchow -- I'm not sure when Steve 18 Muchow left. Peter Bird, I'm not sure when Peter Bird 19 left. They were levels above him. I'm not sure, I'm 20 afraid. 21 Q. Okay. Can we look at paragraph 7 of your witness 22 statement, please, which is at the foot of page 2. You 23 say in the second sentence: 24 "The team [that's the SSC] does not support the 25 hardware or operating systems. The team had a good 7 1 interaction with the testing teams and development to 2 supply evidence and find possible ways to recreate 3 defects on test equipment. We also interacted with 4 subpostmasters when gathering evidence or providing 5 support. The ... SSC was not responsible for reporting 6 to Post Office." 7 Who was responsible for reporting to Post Office? 8 A. I know that Mik did do monthly reports. 9 Q. And Mik -- 10 A. So Mik Peach did monthly reports up to his management. 11 There was also service management -- 12 Q. Sorry, just stopping there. You say that he, Mik, did 13 reports up to his management? 14 A. Yes. 15 Q. Was that still within Fujitsu or ICL? 16 A. Within Fujitsu. He also -- I'm not sure of the date 17 totally. He introduced something called the SMP, 18 service management portal, which he -- 19 Q. Can you explain what the SMP was? 20 A. So it was a website that Mik introduced and wrote and it 21 was for him to put reports on and I believe the change 22 management OCPs were also copied onto there and that was 23 for Post Office to have visibility of these. 24 Q. Did Post Office have direct access to the SMP? 25 A. Yes. 8 1 Q. You were about, in your first answer, to go on to speak 2 about the service management team? 3 A. So, yes, service management was really the interface, 4 I believe, between support issues and Post Office. 5 Q. Just stopping you there, where were they based? 6 A. They were in -- they're Fujitsu. I think they were in 7 Bracknell, as well. And then I was going to talk about 8 the MSU, the management support unit. They did the 9 reconciliation and they reported -- 10 Q. The reconciliation of what? 11 A. Sorry, if there were any reconciliation incidents, so 12 they would then report those reconciliation incidents 13 back to the Post Office. 14 The term I remember currently is BIMs, business 15 incident management, but there is also -- reading 16 through PinICLs, some red but I don't know what red 17 represented. 18 Q. You say in paragraph 25 of your witness statement: 19 "To the extent that there were any known defects 20 when releases were rolled out, my understanding is that 21 this would have been communicated to Post Office, either 22 by the Service Management team ... or by other ICL ... 23 teams. I was not involved in communications with Post 24 Office in this regard, neither am I aware of how or if 25 such issues were communicated to subpostmasters." 9 1 Later in your statement, in paragraph 47, in 2 relation to the accuracy and integrity of data recorded 3 and processed on the system, you say: 4 "I cannot comment on how general issues would be 5 relayed to Post Office but, in respect of individual 6 incidents, I believe this information was passed back to 7 the Post Office through the BSU/MSU or Service 8 Management." 9 What's the basis for those understandings and 10 beliefs that you give? 11 A. So the first one was about projects, so when we have new 12 functionality entered into the system, it is normally 13 entered in via project. It is not normal support at 14 that stage and projects have a -- projects are managed 15 and, I believe, they are fed back through the project 16 management chain, that -- 17 Q. Yes, and what was the basis for that belief? 18 A. I have been involved in some projects. 19 Q. I'm talking about this one. 20 (Pause) 21 It is paragraph 25, so when known defects -- when 22 the releases were rolled out your understanding that 23 this would have been communicated to the Post Office. 24 I'm asking you for the basis for that belief, please? 25 A. Just because projects reported back. Sorry, I've got 10 1 nothing more than that. 2 Q. So it's a general understanding that that's what should 3 happen -- 4 A. Yes. 5 Q. -- between a service provider, Fujitsu, and its client, 6 the Post Office? 7 A. Correct. 8 Q. You haven't got any actual knowledge of whether that did 9 happen? 10 A. I've got no actual knowledge. 11 Q. You see, we have heard some evidence in the Inquiry 12 that, because this was a PFI -- Public Finance 13 Initiative -- framework, under which the services were 14 being provided, the Post Office had what was described 15 as limited or partial visibility of the design approach, 16 the development approach and defects. Were you aware of 17 that or not? 18 A. Not particularly, no. 19 Q. In relation to the comment in paragraph 47 where you say 20 "I believe that information", that's general issues -- 21 sorry, specific individual incidents, you believe that 22 information was passed back through the BSU/MSU or 23 service management. 24 Again, what's the belief for that, or the basis for 25 that belief and understanding? 11 1 A. So if there was an issue that was a new issue, that 2 would be put into the monthly reporting by the SSC 3 manager and service management were involved in 4 resolution of issues. They were the ones who did the 5 reporting. The BSU is, if it's a reconciliation 6 incident, they would do the reporting. 7 Q. Do you know from personal knowledge the extent of the 8 reporting by MSU/BSU? 9 A. I'm sure in the court case there was a -- released 10 a monthly service management report. I can't remember 11 which incident it referred to, but it had broken down 12 about recent issues. 13 Q. So the court case you're referring to is? 14 A. The GLO, sorry. 15 Q. What's your knowledge of the GLO that you're referring 16 to there? Are you referring to the judgment, or -- 17 A. There was some evidence released as part of the GLO and 18 that included a monthly report from the Fujitsu service 19 management team. 20 Q. So the "evidence", whose evidence are you referring to? 21 A. I couldn't tell you whose evidence it was. 22 Q. I'm just trying to explore where you are getting this 23 knowledge from. Is it as a result of -- 24 A. Yes, I viewed this document that was part of the 25 released documents part of the GLO. 12 1 Q. Sorry, I'm just going to press you a little further. 2 A. Yes, sure. 3 Q. You viewed a document that was included as evidence in 4 the GLO? 5 A. That was released. I was following the GLO case and one 6 of the documents in there was -- that was released as 7 part of the evidence was a service management report. 8 Q. How were you following the GLO? 9 A. We followed the Twitter feed and also there were some 10 solicitors -- we provided some information to the 11 solicitors. 12 Q. So the thing you're telling us about now is based on 13 reading a Tweet about the conduct of the GLO? 14 A. And seeing a document that was from that. 15 Q. Sorry, and seeing a document? 16 A. There was a document that was released as evidence which 17 was a service management monthly report from Fujitsu to 18 Post Office. 19 Q. Okay. Can I move on to helpline systems, please, and, 20 as the first witness who is giving evidence to the 21 Inquiry about support services available to 22 subpostmasters, I would like to use you, please, just to 23 confirm the various levels of ICL and Fujitsu support 24 that were available. 25 I think it is right that, initially, there were 13 1 three levels of support and then that grew to four; is 2 that right? 3 A. Yes. 4 Q. Was the first line of support the subpostmasters initial 5 point of contact -- 6 A. Yes. 7 Q. -- and, essentially, Fujitsu's gateway to the remainder 8 of the service support? 9 A. Yes. 10 Q. Was that carried out by the Horizon System Helpdesk 11 which was later known as the Horizon Service Desk? 12 A. Correct. 13 Q. Would this be a fair summary: it would seek to resolve 14 basic queries and then pass on those that it couldn't 15 rectify to the second line of support? 16 A. Yes. 17 Q. Initially, did the Horizon System Helpdesk people work 18 in Feltham? 19 A. Yes. 20 Q. Was that where you worked -- 21 A. Yes. 22 Q. -- as part of the SSC? 23 A. Yes, Feltham A1. 24 Q. I'm sorry? 25 A. Feltham A1. There are multiple Fujitsu buildings in 14 1 Feltham. 2 Q. I think you say in your statement that it was in fact in 3 the same room as you; is that right? 4 A. That's right. there was a custom built room for AGL, 5 which brought the parties together. So in the same room 6 we had the HSH, we had us, the SSC, the EDSC, we had the 7 operations team and we had GiroBank. 8 Q. And how many of them were there, say, at 2009/2008? 9 A. Just a couple. 10 Q. Just two? 11 A. Right at the beginning, 1997 -- 1996 to 1997, only 12 a couple, very, very limited. When we moved into 13 Bracknell and they moved out, I don't know how many 14 there were then. 15 Q. Did they move to Bracknell? 16 A. Sorry, they moved to Stevenage. 17 Q. Wasn't that the second line of support that moved to 18 Stevenage? 19 A. The second line were also in Stevenage. 20 Q. So, just to make it clear, first line of support also 21 moved to Stevenage; is that -- 22 A. Correct. 23 Q. When was that? 24 A. I'm presuming it was when we also moved out in 1997 but 25 I would have to ask and check. 15 1 Q. The second line of support for software, was that 2 provided by the system management centre, or SMC? 3 A. Correct. 4 Q. Would this be a reasonable description of it: it sought 5 to resolve technical problems itself and acted as 6 a gatekeeper and filter to the third line of support? 7 A. Yes. 8 Q. It was also involved in identifying system events that 9 could indicate a software problem had arisen? 10 A. Yes. 11 Q. There was also, is this right, another second line of 12 support for hardware, as opposed to software? 13 A. Yes. The engineers -- I wasn't very much involved in 14 the engineering. Oh, unless you're talking about the 15 ops team -- no, the hardware would be engineering. 16 Q. They initially worked in Feltham, is that right, the 17 system management centre? 18 A. I don't think they were in place when we were in 19 Feltham. 20 Q. Okay. So what, they only ever existed in Stevenage? 21 A. Correct. 22 Q. The third line of support, I think -- is this right -- 23 provided by a variety of teams depending on the issue, 24 the first of them was you, the system service centre or 25 SSC, and that had, as its focus, investigation and 16 1 rectification of software problems? 2 A. Correct. 3 Q. There was the management support team or management 4 support unit, MSU. That monitored and managed 5 reconciliation errors? 6 A. Yes. 7 Q. A reference data team, were you aware of them? 8 A. I was. They eventually joined into the SSC. 9 Q. Did they focus on errors or problems in or with the 10 reference data upon which Horizon relied? 11 A. Yes. 12 Q. Then operational services division, which I think you 13 called operations, they provided support to network and 14 central system incidents? 15 A. Yes, yes. They looked after the data centres, yes. 16 Q. Then the fourth line of support involved development 17 teams that would make changes to Horizon coding to 18 resolve identified errors, bugs and defects; would that 19 be right? 20 A. Yes. 21 Q. Would you agree that your part of the third line of 22 support, it's intended purpose and functions were to 23 provide a support service to resolve technical problems 24 in the minimum time possible and the minimum disruption 25 to the service and to the network? 17 1 A. Yes. When you say "network", you don't mean physical 2 network, you mean as in ... 3 Q. The system. 4 A. Yes. 5 Q. To provide a centre of technical expertise for customer 6 services more generally, providing technical advice, 7 guidance and expertise -- 8 A. Yes. 9 Q. -- and to maintain the KEL database? 10 A. Yes, we ran the KEL database. 11 Q. Would you agree that the SSC was at the heart of the 12 support services provided for Horizon? 13 A. The software support services, yes. 14 Q. In particular, it occupied a central position in the 15 investigation of bugs, errors and defects? 16 A. Yes. 17 Q. If you look at page 19 of your witness statement at 18 paragraph -- sorry, paragraph 17 of your witness 19 statement, on page 7, about six lines in, you say: 20 "If first line support could not resolve the issue 21 and it was related to the software, it would be 22 escalated to the second line support team." 23 Do you see that sentence? 24 A. Yes. 25 Q. Can you assist us, how would someone in the first line 18 1 of support on the end of the phone know that an issue 2 that was being reported to them by a subpostmaster was 3 or was not related to software? 4 A. I didn't work in the HSH but I believe they had scripts 5 to follow, which would help them. 6 Q. So a postmaster phones up and says "I've got this issue, 7 there's a reconciliation problem", how would the first 8 line support know that that related to software? 9 A. As I say, I did not do their role. However, I do 10 believe they had scripts to follow which they would ask 11 them to check various things throughout the script. 12 Q. I'm going to press you a little bit further because of 13 what you said in your statement. 14 A. Yes. 15 Q. Having gone through the script, how would the first line 16 support know that the issue related to software and 17 therefore pass it to the second line? 18 A. I presume that they would get to the end of the script 19 and it hasn't resolved the issue and then they would 20 pass to the second line team. 21 Q. So it must relate to the software? 22 A. It must not always relate to the software but, because 23 the script will only test so many things -- 24 Q. What training did the first line support have to make 25 decisions about whether an issue related to software or 19 1 did not? 2 A. I couldn't tell you what the training of the first line 3 was. 4 Q. Were, to your knowledge, subpostmasters told that there 5 were three and then four possible lines of support? 6 A. I don't know what the subpostmasters were told about the 7 support hierarchy. 8 Q. You don't know what they knew? 9 A. I don't know what the subpostmasters knew. I know that 10 quite often one of them would talk to us, but I don't 11 know if they knew what role we were providing. I think 12 they would ask for people by name. 13 Q. The subpostmasters would? 14 A. Yes. There was definitely some PinICLs were 15 a subpostmaster who has been talking to someone in third 16 line support would ask could they talk to that person 17 again. 18 Q. Yes, so they have had some dealings with them -- 19 A. Correct. 20 Q. -- they would say "Can I speak to John again please?" 21 A. Exactly. 22 Q. But they wouldn't know when they're phoning up "I've got 23 a problem with software, I need to speak to John"? 24 A. No, no idea. 25 Q. With what frequency would software issues, to your 20 1 knowledge, be referred to second line support? 2 A. I couldn't tell you, but I'm sure from the PowerHelp 3 tickets, you could work it out because they've got the 4 team transfers in the PowerHelp. I could tell you that 5 about 2 per cent of calls came from PowerHelp to PinICL 6 and about half of those were raised by subpostmasters, 7 so about 1 per cent of calls were raised by 8 subpostmasters to the SSC, but I couldn't -- 9 Q. And the other 1 per cent? 10 A. The other one was BSU reconciliation -- sorry, issues 11 passing from other teams, not necessarily the 12 subpostmasters, but SMC or BSU. 13 Q. Why were the teams split up? 14 A. Why were the HSH and SMC split up or? 15 Q. Yes. 16 A. I presume that the SMC -- 17 Q. Don't worry about presumptions or speculation; do you 18 know? 19 A. I don't know. 20 Q. If you don't know an answer to a question it's best to 21 say it -- 22 A. Okay. 23 Q. -- rather than put together maybe fragments of evidence 24 and to speculate. 25 A. Okay. 21 1 Q. Were you party to any discussion over whether the 2 support teams should remain together, rather than 3 splitting up into different offices? 4 A. No. 5 Q. Was there, within third line support, ever discussion 6 over trends or patterns that emerge from the nature of 7 calls that were being received, for example a theme is 8 emerging that there are constant problems with 9 balancing? 10 A. Definitely would look at trends and investigate things. 11 If you never got quite to the bottom of something, you 12 saw something again, you would continue. You would 13 normally raise a KEL on a topic, and then you would say 14 on there, you know, "If this happens again could you 15 please examine this and this". Sometimes evidence was 16 too old by the time we got there. 17 Q. What do you mean by that "sometimes the evidence was too 18 old"? 19 A. Sometimes the evidence had been archived away. 20 Q. Archived by who? 21 A. By Riposte. 22 Q. What difficulty did that present? 23 A. It meant that you could sometimes not get to the bottom 24 of an issue so you would raise a KEL and, if it occurs 25 again, then you know where to look at straight away. 22 1 Q. When you say it had been archived away by Riposte, was 2 that a function of Riposte that could not be broken into 3 or interfered with? 4 A. Archiving definitely could be changed, yes, and, 5 actually, there were features to turn archiving off if, 6 for example, the system had been off for a long time 7 but, yes, archiving could be changed. 8 Q. That's a separate issue, whether archiving could be 9 changed. In respect of data that had been archived, was 10 it impossible to look at it? 11 A. It wasn't impossible because it would have gone to 12 audit, but -- yes, so you could have got information 13 from audit. 14 Q. You said that it was difficult sometimes because Riposte 15 had archived the material. Did you ever -- or were you 16 ever a part of a process to obtain material from 17 archive, in order properly to investigate an issue? 18 A. We definitely made a request to the archive team, yes. 19 Q. So that was a theoretical difficulty rather than 20 an actual one; would that be right? 21 A. Yes. Sorry, I was trying to come up with reasons why 22 you may not have got to the bottom of a problem. 23 Q. Yes, and why were you trying to come up with reasons why 24 you might not have got to the bottom of a problem? 25 A. Because you were asking about how you may -- the process 23 1 for going around to documenting a trend. 2 Q. Yes, and so this is a theoretical obstacle that could be 3 overcome? 4 A. That one was. 5 Q. If you wanted to get to the bottom? 6 A. Yes. 7 Q. What other obstacles would there be in getting to the 8 bottom of a problem? 9 (Pause) 10 A. I'm going to have to look at some PinICLs or KELs and 11 come back on that. 12 Q. I'm sorry? 13 A. I would look at some PinICLs and KELs and come back to 14 you about reasons why we have raised some to trend 15 analysis, if that's okay. 16 Q. Does it follow from the need to carefully think about it 17 that there's nothing obvious that strikes you -- 18 A. There's nothing obvious, yes. 19 Q. -- that prevented, other than the very theoretical thing 20 that you have mentioned, in getting to the bottom of 21 a problem? 22 A. Yes. 23 SIR WYN WILLIAMS: Mr Beer, could the statement be taken 24 down from my screen? 25 MR BEER: I'm so sorry, sir. Yes, of course. 24 1 SIR WYN WILLIAMS: Thank you. 2 MR BEER: Was the main mechanism for picking up themes the 3 use of the KEL system? 4 A. Not particularly. The KEL system was very useful for 5 SMC with eventing. It was useful to see if this issue 6 had occurred before but, generally, if -- things 7 occurred before you tended to know them, so it was a way 8 of say providing advice and guidance on how to deal with 9 something, mainly if you have not seen it very often. 10 Q. What was the mechanism, if any, for picking up themes 11 and trends then, if it wasn't the KEL system? 12 A. The KEL system was good because -- sorry, if we had 13 a lot of incidents with the same issues, then if they 14 were actually found to be defects and passed on to 15 fourth line, there would be trends in that because of 16 the number of PinICLs raised and applied to the same 17 products, that you can see in the PinICLs. 18 If the KEL system was good for identifying if 19 something had occurred before as well, we did sometimes 20 add onto it "Could you add other PinICL references if 21 this reoccurs", so there was trending in the KEL system 22 as well. 23 Q. Was there any other system operated, to your knowledge, 24 to pick up themes and trends in the problems with the 25 system that were being reported to Fujitsu? 25 1 A. Not in the SSC. 2 Q. In any other part of the service help levels of support 3 to your knowledge? 4 A. There were other teams like QFP and -- 5 Q. What does QFP stand for? 6 A. Sorry, quality filtering process -- that would manage 7 incidents to the -- so when we passed a ticket in PinICL 8 to the fourth line people, it would often go through the 9 quality filtering process team, who decide where it was 10 to go to, which area of expertise inside the fourth line 11 support teams, and so there was also analysis of when 12 ticket -- working out the amount of effort a fix may 13 take, that that was all in part of the development and 14 release process. 15 Q. That sounds like it is more about systems control within 16 Fujitsu for the benefit of the efficient operation of 17 the help service within Fujitsu. 18 A. Yes. 19 Q. I'm talking about something that's of benefit perhaps to 20 the Post Office or to subpostmasters, ie something 21 within Fujitsu where repeated errors, bugs or defects, 22 or even repeated calls about the same system issue, for 23 example balancing, were picked up to say "Look, we've 24 got a trend developing here, we need to undertake a root 25 cause analysis", or something like that? 26 1 A. There was nothing automated that I know of. 2 Q. What about people? 3 A. Yes, I mean, there were people in the support teams 4 and -- 5 Q. Which part of the support teams? 6 A. Sorry, there was nothing in the SSC that I know of that 7 was -- 8 Q. Had that function? 9 A. -- dedicated to do that function. There was customer 10 service and service management teams that -- 11 Q. What level of the four were they? 12 A. They weren't support teams, sorry, they were the people 13 that I said would report to Post Office the major 14 incidents, and things like that. 15 Q. How would they get to know about any trends or themes 16 that were developing? 17 A. Only if they would be reported up so -- 18 Q. By? 19 A. By, I would say, the helpdesk, or the SMC, or us, the 20 SSC, through management. 21 Q. Did you do that? Did you take a step back? Rather than 22 dealing with the next ticket on the line, did anyone in 23 your team take a step back and say "There's a theme 24 developing here, there's an underlying issue, we need to 25 make a reference"? 27 1 A. I can't give you any examples of that. 2 Q. Can I turn to the Riposte product, please. At page 15, 3 paragraph 48 of your statement, at the foot of the page, 4 you say: 5 "In terms of deficiencies during this time, there 6 were a number of difficulties arising from the Riposte 7 product. These included malformed messages ... and 8 replication issues." 9 What were the difficulties arising from the Riposte 10 product? 11 A. So the malformed messages is when a message is missing 12 attributes, so Mr Cipione broke down what a message 13 attribute -- Riposte message looks like, and it has 14 different attributes in it, and we used to use a system 15 called a TIP repair tool when these messages were 16 harvested into the TPS system, and some of these 17 attributes were missing. Then we would have to go and 18 look and see where -- what was happening on the counter 19 when that message was written to identify what the 20 missing attributes were. 21 Q. What was the cause of the malformed messages? 22 A. I don't know what the underlying root cause of that 23 problem was. 24 Q. Was that ever investigated? 25 A. I'm sure it was. 28 1 Q. By who? 2 A. It would have been fourth line support talking to 3 Escher. 4 Q. Was the cause of the difficulties the coding? 5 A. I don't know what the root cause was. 6 Q. Were you ever told back down the line what the root 7 cause was? 8 A. Sometimes -- if you had a ticket and it was being 9 investigated by fourth line support, you would hold on 10 to a ticket to find out what the root cause was. 11 Q. You tell us in your statement that malformed messages 12 could potentially result in a receipts and payments 13 mismatch but this would unlikely have caused the 14 discrepancy, ie a loss or a gain. How would 15 a receipt -- a mismatch problem or issue, manifest 16 itself to the subpostmaster? 17 A. They were informed by a message saying that there had 18 been a receipts and payments mismatch and it would be 19 when they produced the cash account, the final cash 20 account, I believe. 21 Q. How would the malformed message sometimes cause the 22 discrepancy then? 23 A. The discrepancy -- it could affect the primary mappings, 24 so the -- 25 Q. Sorry, the primary? 29 1 A. Primary mappings, sorry. 2 Q. Can you explain what that is, please? 3 A. So each transaction is added into the cash account using 4 primary mappings. It's like a tree and it builds up and 5 searches for all those transactions that meet that 6 primary mapping, and they are added together to complete 7 that node, and it is all added up together and, if that 8 primary mapping was missing or malformed, then it 9 wouldn't get put into the right place as it builds up 10 the cash account. 11 Q. To your knowledge, was the root cause of those problems 12 fixed? 13 A. I don't know. 14 Q. Do you know what subpostmasters were told when it was 15 suspected that there was a discrepancy caused by 16 a malformed message? 17 A. They would have had the message on screen saying there 18 was a receipts and payments mismatch and then it would 19 have been investigated. There was an event written, 20 I believe, as well, so -- and also harvesting at the TPS 21 database would identify it. So they would -- they could 22 raise a call but, also, we would get the ticket from the 23 MSU/BSU. 24 Q. I'm talking about what the subpostmaster was told 25 themselves, "Look, there's a discrepancy, you've got 30 1 this message" -- 2 A. I don't know what they were -- 3 Q. -- "don't worry, it's not you, you haven't done anything 4 wrong, we believe it's caused by a malformed message"? 5 A. I don't know what the subpostmasters were told. 6 Q. You refer in paragraph 51 of your statement to the fact 7 that: 8 "There could be many root causes for replication 9 failures between counters. This could include network 10 cable faults, hub faults for large branches, hardware 11 faults and issues with Riposte." 12 Can you expand on which of those potential faults 13 were, in your experience, real faults that actually 14 happened in practice? 15 A. I think they all happened in practice. 16 Q. Again, to your knowledge, what were subpostmasters told 17 about this? They get the message that you have spoken 18 about saying that there is a discrepancy, a mismatch; 19 what were they told about the cause of the mismatch if 20 it was attributable to one of these things? 21 A. The replication is different to the corrupt primary -- 22 Q. Malformed message, yes. 23 A. Yes, but the replication would normally be presented to 24 postmasters when they were looking at a transaction, 25 or -- and then it's not there, so run a report and it's 31 1 missing some transactions because they did them on 2 counter 2 and they ran a report on counter 1. 3 Q. Again, can you help us with what they were told about 4 those? 5 A. Again, no, I can't tell you. 6 Q. Is that because it was somebody else's responsibility to 7 tell them? 8 A. It would have come in from the HSH. 9 Q. You said it would have come in from the -- 10 A. When they contact the HSH to report the issue. 11 Q. But they don't know, the subpostmaster, whether this was 12 a hardware fault, they don't know whether it's an issue 13 with Riposte, they don't know whether any of the range 14 of things that you mentioned is a cause of the 15 replication error; all they know is the error message 16 that they're getting. So what process was there to 17 feedback to them, "Look, you haven't done anything 18 wrong, you haven't stolen thousands of pounds here, it's 19 a problem with our system"? 20 A. So if the -- if it was the Riposte one then it wrote 21 an event which was picked up by the SMC and they raised 22 a call and they were contacted -- they contacted the 23 subpostmasters for those. 24 If it was the hardware ones, I don't know. But, 25 again, that wouldn't have caused the receipts and 32 1 payments mismatch. 2 Q. Sorry? 3 A. Again, it was about replication, not corrupted notes. 4 Q. Yes, for the subpostmaster it may not matter 5 particularly, other than to know that it wasn't an error 6 of their own. 7 A. Yes. 8 Q. But you can't help us as to who was responsible for 9 feeding that back to subpostmasters? 10 A. I can't. 11 Q. No, thank you. 12 In paragraph 58 of your statement, you say: 13 "I am not aware of any practices or procedures that 14 may have been in place to obtain input or feedback from 15 subpostmasters during the pilot and rollout of Horizon." 16 Is that because this was a different area of 17 business from you or is it because it didn't happen? 18 A. I couldn't tell you because it was a different area from 19 me. If they contacted -- if a ticket was raised and 20 came to us, we would talk to the subpostmasters relating 21 to that ticket. 22 Q. This is a slightly different issue. This is during 23 pilot and rollout. Were there any problems that were 24 being experienced by subpostmasters, whether there was 25 a mechanism to capture those and to incorporate any 33 1 fixes to them in the system. You're not aware of, kind 2 of, that process? 3 A. No, I'm not. 4 Q. Can I turn to a different issue then, please. For how 5 long have you known Anne Chambers? 6 A. Quite a long time. She joined the SSC -- I can't tell 7 you how long, but it was many years, more than 8 ten years. 9 Q. What was her function in the SSC? 10 A. She was a project specialist. She dealt with counters 11 in particular. 12 Q. Was she there from the start, from your recollection? 13 A. Not from the start but she was there a long time. 14 Q. How closely did you work with her? 15 A. Very closely. 16 Q. Was your contact with her frequent then, on a daily 17 basis? 18 A. Yes. 19 Q. How close did you sit from her, physically? 20 A. A couple of desks away. It was a strange arrangement of 21 desks. 22 Q. What was her role and function when you worked alongside 23 her? 24 A. She was another SSC product specialist. 25 Q. And I think you said specialised in the counters? 34 1 A. Yes, her area of expertise was in the counters. 2 Q. Just explain what specialism in the counters means? 3 A. So when a ticket comes into the SSC, we had 4 a pre-scanner and the pre-scanner's role was to analyse 5 the ticket, check it had all the information expected on 6 it and then route it to a member of the team in the SSC, 7 based on their workload and their areas of expertise 8 and, as I say, she worked on the counter tickets. 9 Q. Did you become aware of her being asked to give evidence 10 in a court case? 11 A. Yes, we were. 12 Q. You say "we were"? 13 A. Yes, the SSC as a whole were aware of this. 14 Q. Can you remember when that was? 15 A. I can't remember the exact date, but I do remember that 16 Anne was unhappy to be asked. 17 Q. She was unhappy? 18 A. Yes. 19 Q. This was before she had actually given evidence; is that 20 right? 21 A. Correct. 22 Q. Can you remember whether there was discussion before she 23 gave evidence about her suitability as a witness or the 24 appropriateness of a member of the SSC going along to 25 give evidence? 35 1 A. I don't know about that conversation. 2 Q. Was there a conversation between you and Anne, or you 3 and other members of the SSC and Anne, about the 4 appropriateness or suitability of her going along to be 5 a witness? 6 A. There was conversations about whether SSC people were 7 the right people to be used. 8 Q. Why was there a question over whether SSC people were 9 the right people to be used? 10 A. I think we thought it was more -- because we were 11 very -- technically specialists in that area and not 12 expert witnesses, we were very unhappy about that 13 process. 14 Q. Was Anne Chambers very unhappy about the process? 15 A. I believe she was. 16 Q. Did she say that to you? 17 A. I cannot recall the conversation, but I believe she was. 18 Q. After she gave evidence, was there any discussion about 19 the appropriateness of her doing so or her suitability 20 as a witness? 21 A. I don't know if there was anything about her suitability 22 but I know that she fed back to the SSC manager that she 23 didn't find it at all nice and we -- I do not believe 24 that -- I believe the SSC manager then pushed back to 25 say -- so that it never happened again. 36 1 Q. The SSC manager that she spoke to was? 2 A. Mik Peach. 3 Q. You said that the SSC manager, words to the effect of, 4 ensured that it never happened again. Who did Mik Peach 5 take that up with to your knowledge? 6 A. I don't know. 7 Q. What was the issue with her giving evidence then? What 8 was the problem about it? 9 A. We just weren't expert witnesses. It was a -- it did 10 not feel right. 11 Q. Do you know why she did it? 12 A. I believe that she was manoeuvred into it. I don't know 13 if she really wanted to do it. She had dealt with the 14 case, I believe. 15 Q. Who was she manoeuvred by? 16 A. I don't know. 17 Q. On what basis do you say that she was manoeuvred? 18 A. I don't think she would have wanted to do it otherwise. 19 Q. Who are the candidates for manoeuvring her into doing 20 it? 21 A. I don't know. 22 Q. Can you help us? 23 A. I would talk to -- about the security teams maybe, who 24 would have interfaced with the request for that. 25 I don't know. 37 1 Q. At what level was her unhappiness at being asked to give 2 evidence and then, after she had done so, expressing her 3 unhappiness about having done so? 4 A. On a scale of -- 5 Q. Yes, of mildly fed up at the bottom end, to incandescent 6 with rage at the top end, say? 7 A. She was probably in the middle. She was really -- said 8 how unpleasant it was and she did not want to do it 9 again. 10 Q. For how long have you known Gareth Jenkins? 11 A. Gareth, I think, was there from the beginning. I recall 12 seeing him in Feltham, so it would have been from 13 probably 1996. 14 Q. How closely did you work with Mr Jenkins? 15 A. So we interfaced quite a bit about -- he was the fourth 16 line and -- so the development and architecture, and he 17 was a specialist in the Riposte area, so if we had some 18 issues in that area we would talk to him. He was 19 approachable. 20 Q. How frequent was your contact with him? 21 A. Maybe monthly. 22 Q. Would that be face-to-face or via emails? 23 A. Normally emails or PinICLs. 24 Q. Did you have meetings with him? 25 A. I have definitely been in meetings with him. I think 38 1 one of my witness ones is meeting with him. 2 Q. To your knowledge, what was his function? 3 A. He was either chief technical or he was one of the 4 technical -- chief technical people, architects, for the 5 Riposte area and, later on, he was also in HNG-X. 6 Q. Were you aware of any discussion about the suitability 7 of him or the appropriateness of him as a witness to 8 give evidence? 9 A. Not until the GLO. 10 Q. So after the event -- 11 A. Yes. 12 Q. -- when you saw that issue emerge in the course of the 13 Group Litigation? 14 A. Yes. 15 Q. Was there any contemporaneous discussion that you are 16 aware of as to the selection of an appropriate witness 17 to give evidence, either in written form or orally, in 18 criminal proceedings against subpostmasters for theft or 19 false accounting? 20 A. Not that I'm aware of. 21 Q. We are aware of an article in Computer Weekly, a trade 22 journal, of 11 May 2009. Can you remember when you 23 first became aware of that? 24 A. In this -- I think you mentioned it recently. 25 Q. That's the first you have known of the Computer Weekly 39 1 article? 2 A. 2009, when I have watched some previous articles in -- 3 on the online Computer Weekly about things. 4 Q. Does it follow that the Computer Weekly article of 5 May 2009 wasn't discussed in the office at about the 6 time that it came out? 7 A. I don't recall that. 8 Q. When you say you don't recall it, that could mean that 9 it may have happened but you may have forgotten, or 10 "I don't recall it because it is likely that it didn't 11 happen"? 12 A. I don't recall it. It could have happened but I do not 13 recall a conversation about it. 14 Q. Can I turn to a separate issue, please, the issue of 15 remote access. 16 Could we look, please, at POL00030029. It will come 17 up on the screen for you. 18 A. Thank you. 19 Q. Can we look at page 4, first, please. At the foot of 20 the page this is an email of 13 May 2014, from Sean 21 Hodgkinson. If we just look at the bottom of the next 22 page, please, to see who he was: senior consultant in 23 the audit advisory division of Deloitte, yes? 24 A. Yes. 25 Q. Then if we just go back to where we were, please, the 40 1 previous page. Thank you. You can see that the email 2 of 13 May 2014 is to a range of people. You are not 3 included on this chain but, as we will see in a moment, 4 you end up answering the questions in this chain. Do 5 you remember? 6 A. I do, from reading. 7 Q. Yes. I just want to see what the questions were first 8 and this is to a collection of people, I think 9 substantially within the Post Office: 10 "All, 11 "Following review of the technical design document 12 in relation to the Branch Database, I had a couple of 13 queries that I was hoping you may be able to help with. 14 If not, please could you direct me toward somebody who 15 may be able to assist: 16 "1) Balancing Transactions. 17 "Section 5.6.2 ..." 18 Do you know what that is of? 19 A. No. 20 Q. "... describes back end database amendment process which 21 is included by design ..." 22 Then he quotes from the document "Inserting 23 Balancing Transactions": 24 "There is a requirement that the SSC will have 25 ability to insert balancing transactions into the 41 1 persistent objects of the Branch Database. There are 2 reasons for SSC having to do so, eg to rectify erroneous 3 accounting data that may have been logged as a result of 4 a bug in the Counter/BAL." 5 Over the page, please: 6 "SSC will have privileges of only inserting 7 balancing/correcting transactions to relevant tables in 8 the database. SSC will not have any privileges to 9 update or delete records in the database. Any writes by 10 the SSC to BRDB ..." 11 BRDB? 12 A. Branch database. 13 Q. "... must be audited. The mechanism for inserting 14 a correction record must ensure that the auditing of 15 that action performed must be atomic." 16 What do you understand that to mean? 17 A. So "atomic" is a database terminology, so you write all 18 the transactions or they all roll back. You don't have 19 partial transactions written. 20 Q. "There also needs a level of obfuscation to ensure that 21 the audit mechanism is robust." 22 What do you understand that sentence to mean? 23 A. No idea. 24 Q. "The above-mentioned requirements suggest that there is 25 a need for a correction tool to be delivered which 42 1 performs the correction, audits it and saves both 2 changes. 3 "A simple low-cost solution for the tool is to 4 provide a Linux shell based utility ..." 5 Can you help us with what Linux was please? 6 A. It's an operating system that they have used on -- well, 7 on the branch database. 8 Q. "... which calls a PL/SQL package ..." 9 Can you explain what that is, please? 10 A. A programme language SQL is a way of writing structured 11 query language transactions to an SQL database, which is 12 the branch database is. 13 Q. "The package will allow inserts to the following 14 transactional tables in the Branch Database Live schema 15 with the exception of the Message Journal. All inserts 16 will be audited in the table", and then a reference is 17 given. 18 Then the question that Mr Hodgkinson asked: 19 "From the above we wish to clarify, with evidence 20 where possible: 21 "How does this process operate and who has the 22 ability to be able to perform this, eg POL and/or 23 Fujitsu?" 24 Then secondly: 25 "What monitoring is performed over the table", and 43 1 then the reference is given. 2 If we can go back, please, to page 4, we can see 3 Dave King's response. He was the senior technical 4 security assurance manager. What part of the Post 5 Office was that within, to your knowledge? 6 A. I don't know. 7 Q. So this is still within the Post Office at the moment, 8 and he says: 9 "... I believe the only way we will be able to 10 resolve this is if you get confirmation from Fujitsu of 11 whether this has ever been done and what the process is 12 (POL have no direct access to the database)." 13 Does that sentence in the brackets there correspond 14 with your understanding, that POL had no direct access 15 to the database? 16 A. Yes. 17 Q. "If corrections are needed, 'we' insert a transaction to 18 correct the situation following a reconciliation process 19 rather than make direct changes to any transaction in 20 the database." 21 Then raises an issue about a contact within Fujitsu. 22 Can we go back to page 1, please, of the email chain 23 and then if we go to the foot of the page -- keep going, 24 keep going. Thank you. 25 At the very foot of the page we can see an email 44 1 from you to James Davidson of 15 May 2014. Who was 2 James Davidson? 3 A. I don't know. I was asked by someone to provide some 4 technical input from a couple of questions, so I did. 5 Q. You say: 6 "... we did not discuss timescales but I have just 7 been asked by Leighton for some more details before 8 a 10.30 meeting today." 9 Who was Leighton? 10 A. I can't remember, I'm afraid. 11 Q. At this stage, you're saying "I have just been asked by 12 Leighton for some more details before a 10.30 meeting", 13 and it is 10.24 when you are writing the email. Did you 14 have sufficient time to prepare the answers or are you 15 hinting that you hadn't? 16 A. I probably was hinting that I have been given a very 17 tight deadline, so I have not researched this 18 information as thoroughly as I probably could. 19 Q. Did you know what the answers that you were giving were 20 going to be used for, ie the purpose to which they were 21 going to be put? 22 A. No. I was very surprised to read the Deloitte -- 23 Q. I'm sorry? 24 A. I was very surprised to read the Deloitte -- the 25 references in there to this email. 45 1 Q. Why were you surprised of what became of the answers? 2 A. Because I was just asked a couple of technical 3 questions. I mean, I don't mind the answers being 4 there, but no one told me where they were going to go. 5 Q. What, if anything, would you have done differently if 6 you knew where the answers were going to go and what use 7 was going to be made of them in the future? 8 A. I would have missed the 10.30 deadline. 9 Q. What other research would you have undertaken? 10 A. I would have talked to the database -- the database 11 architect. 12 Q. Who was that? 13 A. Gareth Seemungal. 14 Q. Say that again please? 15 A. Gareth Seemungal. 16 Q. So if we look then, question 1, about the -- and then 17 there's a reference to the table -- and then you have 18 broken down the question, part 1: 19 "How does this process operate and who has the 20 ability to be able to perform this, eg POL and/or 21 Fujitsu?" 22 What did you understand the question to mean? 23 A. It's talking about the branch transaction correction 24 utility, and so I was trying to -- I know it has been 25 used once, so I was using that information to try and 46 1 detail what was the process, how that time had come 2 about. 3 Q. You answer it as follows: 4 "The normal support route is used to identify when 5 a fix is required either from a branch raised incident 6 or estate monitors that alert support staff. 7 "An TfS incident would be raised with evidence." 8 What does a "TfS incident" mean? 9 A. So TRIOLE for Services is the first line helpdesk used 10 at this time. 11 Q. Who would raise that incident? 12 A. So that would be -- it depends on where the issue was 13 identified. It could have come from the branch -- MSU, 14 it could have come from a postmaster or from SMC, or 15 from -- in Post Office. 16 Q. You say: 17 "This would be transferred to the SSC as a PEAK 18 because they support the applications." 19 Who is the "they" in that sentence? 20 A. SSC. 21 Q. "The SSC would investigate with evidence from the 22 support branch database and then liaise 4th line 23 development (evidence and progress would be recorded on 24 the PEAK). 25 "4th line development would generate the required 47 1 scripts using a test system to make the correction. 2 An MSC ..." 3 What's an MSC? 4 A. Managed service change, so it's part of the authorised 5 changes to systems. We used to have OCPs and then it 6 became MSCs and now it's TfS, and they're all changes. 7 Q. Overall, what is that describing, an MSC or -- 8 A. It's going to describe what the change is and it's going 9 to go to people to be authorised. It's going to -- this 10 goes to a distributed list who have to authorise it. 11 Q. So: 12 "An MSC ... would be raised for permission to run 13 the support tool on the live branch database. 14 "The SSC would run the script using the support tool 15 against the live estate." 16 So, overall, in this part of the answer, you're 17 describing who has the ability to perform the function 18 and it is generated by either subpostmasters, through 19 first line support, or somebody within Fujitsu 20 themselves. It's picked up by third line support and, 21 if it's necessary to run scripts using a test system, 22 a request would be raised for permission to do so? 23 A. Yes. 24 Q. Is that a fair summary? 25 A. That's a fair summary. 48 1 Q. The second part of the question that you have broken 2 down: 3 "What monitoring is performed over the table ..." 4 Can you explain, first of all, what the question 5 means, "What monitoring is performed over [that] table"? 6 A. That table is the journal that this tool writes to, so 7 I'm presuming it was meaning how is that table populated 8 and then does it go anywhere else, audit or whatever. 9 Q. You answer: 10 "The Support tool is written to run under the SSC 11 (read only role) ..." 12 What does that mean? 13 A. So the roles -- it doesn't have permission to write to 14 database. 15 Q. "... and connects internally as the APPSUP role (write 16 permission)." 17 What does that part of the sentence mean? 18 A. That's the database role that does have permission to 19 write to the database. 20 Q. What does "and connects internally" mean? 21 A. It means that we don't manually have to switch the role 22 to APPSUP. The tool does it all internally. If we 23 needed to switch role to APPSUP we have to request that 24 permission from the SecOps team and the SecOps team get 25 the ops team to make the change and then we can switch 49 1 role to APPSUP. 2 Q. What was "APPSUP"? 3 A. APPSUP is the role that allows write to -- update to the 4 database. 5 Q. What does "APPSUP" refer to? 6 A. Application support, I presume, but that's ... 7 Q. Why would operational security ordinarily be required to 8 be contacted to give permission to use APPSUP? 9 A. So this was a security -- an additional check to make 10 sure that the reason we're requesting write update to 11 the database is reasonable. 12 Q. But this allowed an automated access to the APPSUP role? 13 A. Correct, so normally APPSUP would be -- we would use 14 APPSUP when there is no tool -- tooling defined for 15 it -- for when there is no plan. This is a planned 16 tool. This tool can do all the connections underlying. 17 Q. You say: 18 "All changes are written to the AUDIT logs." 19 What do you mean by that answer? 20 A. I believe that the output from the tooling is written to 21 a log and then that log is written to the audit 22 database. 23 Q. You say: 24 "The output from the support tool is captured and 25 recorded on the PEAK." 50 1 A. That's -- yes, we did that as well, but there's -- that 2 is a manual process. 3 Q. But you're saying that there's a footprint of the use of 4 the tool written automatically to the audit log? 5 A. Correct. 6 Q. "I can find just one recorded use of this tool", and 7 then you set it out. 8 A. Yes. 9 Q. Then over the page, please, you say: 10 "This indicates that this parameter has not been 11 changed since created on [5 October 2009]." 12 A. I think that was going from there is no update time 13 stamp but there is a creation time stamp, that's what 14 I was going from there. 15 Q. What do you mean by "This indicates that this parameter 16 has not been changed"; what are you referring to, the 17 parameter? 18 A. It would be a specific question about a database 19 parameter and that is the output of my query against 20 that parameter: what are the fields on that database 21 parameter? 22 Q. What are you saying by that sentence? 23 A. So I'm detailing the settings of that parameter and 24 making an observation that I believe it hasn't been 25 updated since creation. 51 1 Q. You're saying it has only been used once? 2 A. No, sorry, that is a separate query to the other. There 3 was two queries. One was about the actual tooling and 4 has it been used and then there's another query about 5 this parameter. 6 Q. Yes, if we just go back to the foot of the previous page 7 and up a little bit, it's the bullet point in bold: 8 "Can we see evidence to demonstrate that this 9 parameter is currently set to 'True'?" 10 What does that question mean? 11 A. I'm unaware. I was looking at what the parameter is in 12 that data -- so that question is -- sorry, "There is 13 a parameter in the database, it's in this table, can you 14 find out is the value true?" 15 Q. What does that mean though? 16 A. I -- how that parameter is used, I cannot tell you. 17 Q. You just wanted -- you answered the -- 18 A. I answered the question, the absolute question: "What is 19 that parameter set to?" 20 Q. Overall do the answers mean that the only way that 21 someone in the SSC could amend cash accounts was by 22 using the process that you described or were you saying 23 that that's just one type of process for amending cash 24 accounts? 25 A. Overall, I was answering the question about the usage of 52 1 that tool, which was the question. I would say there is 2 the ability of direct access, but that is extremely 3 difficult. That is the reason why there is a tool for 4 doing such, and why -- there's many tables that are 5 written to in the branch database, not just a central 6 database table with the branch details -- the cash 7 account details, or the BTS details in this time, and 8 you have to update all the correct tables in the right 9 order or atomically, and this is a tool that is designed 10 for that and the -- actually, the fourth line team would 11 devise the scripts to be executed to do it correctly. 12 Q. Would it be wrong to say that, overall, from this email, 13 you were saying that cash accounts have been amended 14 only once? 15 A. I think it is a fair statement because I think of how 16 difficult to update a cash account -- a branch trading 17 statement in HNG-X database is. 18 Q. So that would be a fair statement: you were saying that 19 cash accounts, to your knowledge, had only been amended 20 the once and that was referring to the entirety of the 21 period of time that you had worked in the SSC? 22 A. We're talking about the branch database, we're talking 23 about HNG-X from 2010 to now. 24 Q. Yes. 25 A. Yes. 53 1 MR BEER: Sir, that would be an appropriate moment for the 2 morning break. 3 SIR WYN WILLIAMS: Very well. 4 Can I just ask, Mr Simpkins, so that I'm clear about 5 this, so in the last series of questions and answers, 6 from Mr Beer and your answers, you are confining what 7 you say to the time from the rollout of Horizon Online, 8 as opposed to Legacy Horizon? You're not saying 9 anything about Legacy Horizon? 10 A. Correct, this is talking about the branch database, 11 which is only used from HNG-X. 12 SIR WYN WILLIAMS: Fine, I've got it. Thank you, yes. 13 Right, quarter of an hour, Mr Beer? 14 MR BEER: Yes, so 11.30, please, sir. 15 SIR WYN WILLIAMS: Fine. 16 MR BEER: Thank you. 17 (11.15 am) 18 (Short Break) 19 (11.30 am) 20 MR BEER: Good morning, sir, can you see and hear me? 21 SIR WYN WILLIAMS: Yes, I can. 22 MR BEER: Thank you. Mr Simpkins, just one question arising 23 from the last answer you gave. You said to the Chairman 24 that your email should be read in the context of only 25 referring to Horizon Online. 54 1 A. Yes. 2 Q. You said "in relation to the branch database". What did 3 you mean by reference to the "branch database"? 4 A. The branch database is only used in Horizon Online. It 5 wasn't in existence, it didn't exist in Horizon Legacy. 6 Q. That was something maintained by Fujitsu, it wasn't in 7 the branch? 8 A. That's correct, so, yes, the branch database is in the 9 data centre. 10 Q. Thank you. Can we look, please, at POL00029750. You 11 will see that this is a draft Deloitte report of 12 23 May 2014. If we can skip to page 3, please, and then 13 just look at the first couple of paragraphs: 14 "As outlined to us by the Post Office Limited ... 15 litigation team, 'POL is responding to allegations from 16 subpostmasters that the 'Horizon' IT system used to 17 record transactions in POL branches is defective and 18 that the processes associated with it are inadequate 19 (eg that it may be the source and/or cause of branch 20 losses). POL is committed to ensuring and demonstrating 21 that the current Horizon system is robust and operates 22 with integrity, within an appropriate control 23 framework'. 24 "POL is confident that Horizon and its associated 25 control activities deliver a robust processing 55 1 environment through three mechanisms: POL have designed 2 features directly into Horizon to exert control; POL 3 operates IT management over Horizon; and POL have 4 implemented controls into and around the business 5 processes making use of Horizon. Collectively these 6 three approaches of inherent systems design, ongoing 7 systems management and business process control are 8 designed to deliver a Horizon processing environment 9 which operates with integrity." 10 Then further down the page, please: 11 "Deloitte has been appointed to: 12 "consider whether this Assurance Work appropriately 13 covers key risks relating to the integrity of the 14 processing environment, 15 "to extract from the Assurance Work an initial 16 schedule of Horizon Features, 17 "to raise suggestions for potential improvements in 18 the assurance provision." 19 Then it sets out how it is going to do its work. 20 Were you aware that this process was being undertaken in 21 2014? 22 A. No. 23 Q. Can we look forwards, please, to page 38. I have just 24 shown you those initial parts of the document in order 25 that you can understand what the document is and the bit 56 1 that we're going to look at where it falls within it. 2 As part of their assurance work, Deloitte produce 3 an assurance schedule and they say that they: 4 "... present below a schedule of the Assurance Work 5 and sources we have identified which relate to certain 6 groups of Horizon Features." 7 They record an assessment of the level of comfort 8 that POL has over the relevant Horizon feature. Do you 9 see? 10 A. Yes. 11 Q. Then if we can scroll forwards to page 48, please. Can 12 you see under the "Area", "Usage", in the second box 13 down "Branch Ledger transactions are recorded accurately 14 in the Audit Store", as the assertion giving rise to 15 process integrity? 16 The description of the feature of processing 17 integrity is said to be: 18 "Formalised change control approval and monitoring 19 process over usage of Balancing Transactions". 20 The source of that is said to be an email 21 communication from you of 15 May 2014. That's the thing 22 we looked at and "articulating control design around 23 this process", and the "Level of Comfort" that POL are 24 said to have had is "Partial". 25 Then the next row, the "Key Assertion" giving rise 57 1 to process integrity was: 2 "Branch Ledger transactions are recorded accurately 3 in the Audit Store. 4 "Description": 5 "Audit trail monitoring the usage of balance 6 transactions." 7 Again, the same source of evidence. Did you know 8 that your email was going to be used in this way? 9 A. No. 10 Q. What, if anything, would you have done differently in 11 terms of its construction and the contents of it if you 12 had known it was going to be used in this way? 13 A. I think I said earlier I would probably have had a talk 14 to the database architect just to clarify that this 15 is -- my email answered these questions. But I was 16 fairly happy with what I replied to for the two 17 questions that I was asked. 18 Q. So am I detecting this, that it was the narrowness of 19 the answers that you gave -- 20 A. Yes -- 21 Q. -- that if you had known they were going to be used for 22 this purpose you might have added more to them? 23 A. Yes. 24 Q. I take it, therefore, that you didn't discuss with 25 Deloitte the provision of your email or the content of 58 1 the answer? 2 A. Definitely not. 3 Q. Can we look, please -- that can be taken down -- at 4 POL00028070. We are three years on now and another 5 report, also in draft, from Deloitte. If we go again to 6 page 3, please, you will see a summary from Deloitte of 7 the Horizon Online system. It sets out the controls 8 that respond to the fundamental risks under those 9 subparagraphs. 10 Can you recall this report being produced? 11 A. No. I have seen it in my bundle, but I don't recall it 12 being produced. 13 Q. Do you recall whether they, that's Deloitte, spoke to 14 you about it, the contents of the report? 15 A. No. 16 Q. Can we just look forwards, please, to page 83 of the 17 document, please. In an appendix, they set out a list 18 of individuals that they, Deloitte, say were interviewed 19 and can you see your name two from the bottom here -- 20 A. I can. 21 Q. -- "John Simpkins, SSC team leader". Were you 22 interviewed by Deloitte? 23 A. I don't recall being interviewed by Deloitte, no. 24 Q. You would probably remember if you were, wouldn't you? 25 A. I would have thought so. 59 1 Q. So this is incorrect? 2 A. They have also got Jon Hulme as working for Post Office. 3 Q. I'm so sorry? 4 A. Sorry, the one above is incorrect as well. 5 Q. Ie his employer ought not to be POL? 6 A. Is Fujitsu, yes. 7 Q. So, in any event, as far as the content of the 8 October 2017 Bramble report for Deloitte, you were not 9 interviewed for that? 10 A. I don't recall ever being interviewed for that. 11 Q. That can be taken down, thank you. 12 Can we look, please, at FUJ00088036. If that can 13 just be expanded a little bit, please. 14 Do you recognise this? 15 A. Yes. 16 Q. What do you recognise it as? 17 A. It's a support -- well, it's a design document for when 18 we were introducing OpenSSH to remotely access the 19 counters. 20 Q. So we're here dealing with Legacy Horizon, as it became 21 known -- 22 A. Correct. 23 Q. -- not Horizon Online? You would have been, I think, 24 provided with this at the time, or seen it at the time, 25 or had access to it at the time? 60 1 A. We would have had access to it. We -- the SSC were 2 generally on a standard distribution list to comment on 3 documents and give feedback to documents but they were 4 routed out amongst the team. I don't know if the 5 dimensions, or if -- this was probably PBCS(?), I don't 6 know if that contains the reviewer's comments to see 7 who -- 8 Q. If we skip forwards, and then go down, is that what you 9 are referring to, the reviewer's details, ie those that 10 were given the opportunity to review? 11 A. That's correct, yes. So you've got mandatory -- you've 12 got Mik Peach and he was just the figurehead for the 13 document reviews. They would be sent to the SSC and 14 then given to someone. 15 Q. Then Mr Peach underneath him, I think? 16 A. Yes. 17 Q. Sorry, Mr Parker underneath him? 18 A. Yes. 19 Q. Thank you. So this would have been a document that the 20 SSC had an opportunity to review and comment on and 21 then, in its final iteration, distribute it to the 22 members of the SSC? 23 A. No, it would be put in dimensions storage. We may put 24 it onto our SSC website some -- if it were the -- if the 25 final version were sent to us, this is the type of 61 1 document we would put on the SSC website, so it's 2 searchable. 3 Q. So members of the SSC would have access to it? 4 A. Correct. 5 Q. Thank you. Can we just go to page 9, please, and look 6 at the introduction to see what the document is. Under 7 1.1.1, "General": 8 "[SFS] ..." 9 I think that's "security function specification"; 10 would that be right? 11 A. I don't know. 12 Q. If I'm right that that is what SFS means, security 13 functions specification, what was the security function 14 specification? 15 A. I don't know. 16 Q. Anyway it, assuming that it is what I say it is: 17 "... mandates the use of Tivoli Remote Console ... 18 for the remote administration of Data Centre platforms." 19 Can you explain what that sentence is saying, 20 please? 21 A. So Tivoli was a management package that was used for 22 eventing, amongst other things, and had the ability to 23 run some commands, and part of it was a remote console 24 which allows you to commit to a computer in a console -- 25 a command line facility, so you can execute commands on 62 1 that computer. 2 Q. Thank you. It continues: 3 "This records an auditable trail of log-ins to all 4 boxes accessed by the user." 5 Is that accurate, to your knowledge? 6 A. I believe so. I didn't manage Tivoli. 7 Q. It says: 8 "It is a matter of considerable discussion and 9 correspondence that the [Tivoli Remote Console] is slow 10 and difficult to administer." 11 Do you remember that, ie that it was slow and 12 difficult to administer? 13 A. Not particularly. 14 Q. "This has led over time to BOC personnel ..." 15 BOC, can you help us with what that was? 16 A. No. 17 Q. Maybe Belfast Operation Centre? 18 A. Could be. 19 Q. If it is Belfast Operation Centre, what was the Belfast 20 Operation Centre? 21 A. They were the operations people, so -- 22 Q. So part of Fujitsu in Belfast? 23 A. Correct, yes, they looked after the data centres. 24 Q. "... relying heavily on the use of unauthorised tools 25 (predominantly Rclient) ..." 63 1 What was "Rclient"? 2 A. That was a remote client so that's another tool that you 3 can use to get a command line interface onto a server 4 remotely. So that's what -- I remember we did use that 5 to connect to the counters. 6 Q. You used that as well, did you? 7 A. We used that to connect to the counters. 8 Q. To connect to counters? 9 A. Correct. 10 Q. "... to remotely administer the live estate. Its use is 11 fundamental for the checking of errors." 12 Would you agree with that sentence? 13 A. Yes. 14 Q. "The tool does not however record individual user access 15 to systems but simply records events on the remote box 16 that Administrator access has been used." 17 Does that reflect your understanding? 18 A. Yes, so -- yes, you would probably have a Windows event 19 that that user has been granted authorisation to connect 20 to the box, so a security event, I would imagine. 21 Q. But it doesn't record what happened? 22 A. It wouldn't record -- yes. It wouldn't record -- 23 Q. It was fact of access but not -- 24 A. Or even who did it. It would have been under a generic 25 user. 64 1 Q. So it doesn't record what the purpose of the access was 2 or what was done in the course of access and it doesn't 3 record who has access. As you say, it would be 4 a generic record? 5 A. Yes. 6 Q. "No other information is provided including success/fail 7 so it is not possible to simply audit failures. The use 8 of such techniques puts Pathway in contravention of 9 contractual undertakings to the Post Office." 10 Do you remember that issue arising back when using 11 Legacy Horizon? 12 A. Not particularly. I do remember we used Rclient. 13 I don't particularly remember the Tivoli remote console, 14 but I don't remember particularly using it, and then -- 15 Q. Do you remember an issue being raised as to the SSC's 16 use of Rclient putting it in breach of its contractual 17 obligations or undertakings to the Post Office? 18 A. I don't particularly remember that but I do know that we 19 did switch to using OpenSSH to connect. 20 Q. "After proposals in this SOD ..." 21 I'm afraid I couldn't find what that meant: "SOD"? 22 A. The system support -- outline design, that's what -- 23 this document, is it? 24 Q. Ie this very document? 25 A. Yes. 65 1 Q. The system outline design? 2 A. Yes. 3 Q. I've got it. So: 4 "After the proposals in this [document] have been 5 implemented a CP ..." 6 Can you help us with that? 7 A. Change proposal. 8 Q. "... will be raised to phase out [Tivoli Remote 9 Console] ... 10 "This document provides an outline design, which 11 primarily stops Pathway being in contravention of its 12 contractual undertakings but also provides an acceptable 13 and agreed level of secure access to systems for support 14 activities." 15 Can you help us with what, if any, relationship the 16 BOC -- if I'm right, the Belfast Operation Centre -- had 17 to the SSC? 18 A. So they looked after the data centre systems, so the 19 operating system of the data centre servers, the 20 databases in the data centre. So if it wasn't written 21 by Pathway, they generally looked after it; if it was 22 written by Pathway, we looked after it, if that makes 23 sense. 24 Q. I think I understand. Can we go to page 13, 25 paragraph 4.1.2, please. Can we just scroll down 66 1 a little bit. I should read 4.1 first, "Areas of 2 concern": 3 "There are two major areas of concern with the 4 current support processes: 5 "Second line support does not have the tools 6 necessary to perform their function ... 7 "Third line and operational support organisations 8 access to the live system is not fully audited and in 9 some cases is restricted in the actions that can be 10 carried out; 11 "The consequences of these two issues are specified 12 in the following sections." 13 Then under 4.1.2: 14 "Third line support staff receives repeat instances 15 of calls that should have been filtered out by second 16 line. Handling repeat calls is not an effective use of 17 third line support resource. 18 "The current support practices were developed on 19 a needs must basis; third line support diagnosticians 20 had no alternative other than to adopt the approach 21 taken given the needs to support the deployed Horizon 22 solution. 23 "The consequences of limited audit and system admin 24 access afforded to third line support staff provide the 25 opportunity to: 67 1 "Commit fraudulent acts; 2 "Maliciously or inadvertently affect the stability 3 of the new Network banking and Debit Card online 4 services; 5 "In addition a complete audit would allow Pathway to 6 defend the SSC against accusations of fraud or misuse." 7 Again, in 2002, did you know that this was an issue? 8 A. I was unaware that this was an issue. 9 Q. Did you know that an investigation or a review was being 10 undertaken into the extent of third line support access 11 and the method that the SSC was using to procure such 12 access and that it was said to have provided the 13 opportunities set out there? 14 A. Not particularly. I do remember we were talk -- 15 I remember us talking about the OpenSSH access and 16 I also remember it being told that it was going to 17 record every key press. So I knew that there was 18 enhanced audit in what we were moving to but I don't 19 remember particularly that it was put to us in this way. 20 It was -- yes, it was enhanced audit. I did know that 21 was coming in. 22 Q. Can you repeat that last sentence, I didn't hear it? 23 A. It was enhanced auditing and, in this new method of 24 access, I knew that was coming in. 25 Q. So you knew that a new method of access that was more 68 1 auditable -- 2 A. Correct. 3 Q. -- was being introduced, you didn't know the reasons 4 that sat behind it? 5 A. Yes, so, obviously, I can infer something has come in 6 that's more auditable, the old one obviously was not 7 auditable enough. 8 Q. Would you agree with what is said here as to the reasons 9 for its introduction, namely that the type of access 10 that was afforded did give those opportunities? 11 A. I don't know if I agree with the first one. 12 Q. That it didn't give the facility to staff to commit 13 fraudulent acts? 14 A. Yes, I'm -- as far as I'm aware, the APS transactions 15 and banking transactions were all digitally signed. So 16 I can't see how SSC would be able to do any fraudulent 17 activities there. 18 Q. The second one, maliciously or inadvertently -- 19 A. I imagine maliciously, you could try and damage 20 a database or take down an agent which would cause 21 an outage, or VPN server. So yes, I could see 22 maliciously. 23 Q. We can put that to one side. Can we look, please, at 24 FUJ -- 25 I'm so sorry, we should have looked at one other 69 1 passage in that document. 4.3.2 on page 15, please. 2 Thank you. The authors record that: 3 "All support access to the Horizon systems is from 4 physically secure areas. Individuals involved in the 5 support process undergo more frequent security vetting 6 checks." 7 Were those two things accurate? 8 A. Yes. 9 Q. The site was physically secure and there was some 10 enhanced vetting? 11 A. Yes, so we had security checks on all the staff. The 12 site -- the room on the sixth floor had its own pass 13 system. It wasn't part of the general building pass 14 system. The -- we had separate computers for connecting 15 to the data centre, as well as your corporate system. 16 It was on a totally separate system. You had separate 17 passwords. You had two factor authentication with 18 secure IDs. So, yes, it was fairly secure. 19 Q. Then it says: 20 "Other than the above controls are vested in manual 21 procedures ..." 22 That doesn't make complete sense: 23 "... requiring managerial sign off controlling 24 access to post office counters where update of data is 25 required." 70 1 It's difficult to understand exactly what that 2 means. 3 A. It's probably talking about the OCPs and OCRs and the 4 MSCs, and things we were talking about, where there were 5 other sign offs, but that was a manual sign off to give 6 you authorisation, but it didn't physically stop you 7 doing it without that. 8 Q. And there was no audit of it? 9 A. Correct. 10 Q. "Otherwise third line support has: 11 "Unrestricted and unaudited privileged access ... to 12 all systems including post office counter PCs ..." 13 That was true, yes? 14 A. Yes. 15 Q. "The ability to distribute diagnostic information 16 outside of the secure environment; this information can 17 include personal data (as defined by the Data Protection 18 Act), business sensitive data and cryptographic key 19 information." 20 That was true as well? 21 A. No. 22 Q. No? In which respects was it false? 23 A. So we didn't support the KMA -- we didn't support the 24 key management. We supported its interactions, but we 25 didn't support it -- that was where the key material 71 1 was, I believe, and we didn't support the audit server 2 either, so we didn't have access to those. We had -- 3 there was a separate key server, which was in a little 4 room that was locked and used by the security people. 5 There was a KMA work station, which was used by a fourth 6 line support person who did the support for the key 7 management. So there were areas we didn't support. 8 Q. Right, so it's an accurate statement but needs to be 9 qualified, in that there are some areas that it does not 10 apply to? 11 A. Yes. 12 Q. Is that a fair way of describing it? 13 A. Specifically, I'm thinking about the cryptographic key 14 information. 15 Q. Skipping a paragraph, which is a repetition largely of 16 what appeared previously, the authors record: 17 "There are ... no automatic controls in place to 18 audit and restrict user access. This exposes 19 Fujitsu ... to the following potential risks: 20 "Opportunity for financial fraud ..." 21 Would you agree with that? 22 A. No, I don't see how you could do financial fraud. 23 Q. "Operational risk -- errors as a result of manual 24 actions causing loss of service to outlets ..." 25 A. Yes. 72 1 Q. You agree with that? 2 A. Yes. 3 Q. And: 4 "Infringements of the Data Protection Act." 5 A. Yes. 6 Q. You would agree with that, thank you. 7 Now, this process that's being described, ie the 8 backward look and the fixes that were proposed, you 9 didn't include any of that in your email of May 2014? 10 A. No. 11 Q. Is that because you were answering the narrow question 12 that was asked of you? 13 A. There were literally two questions and I answered them 14 both. 15 Q. Can we look at FUJ00089756. 16 A. This also is -- 17 Q. This is Legacy? 18 A. -- Legacy and the questions were in -- 19 Q. They don't say Horizon Online but they could only apply 20 to Horizon Online? 21 A. Exactly. 22 Q. Can we look, please, at -- yes, thank you, we've got it 23 up. 24 This is a PEAK, PEAK number 0208119. You will see 25 if we just scroll down a little bit, please, and a bit 73 1 more, that it's opened in February 2011? 2 A. Yes. 3 Q. I think you were aware of this PEAK because it related 4 to your work and, at one stage, I think it was referred 5 to you and you made a contribution to it. I think we 6 can just see that if we go forward to page 3 and just 7 scroll down. I think we can see an entry on there of 8 17 August 2011 by you. Yes? 9 A. Yes, so this is about the APPSUP. 10 Q. Yes, so if we just go back to the beginning then, 11 please, page 1, and the summary of the incident we can 12 see is that: 13 "SSC Database users do not have correct 14 permissions." 15 Can you see whether this was raised by somebody 16 within Fujitsu or -- 17 A. Yes, it is "Call Logger", top right, by Mark Wright in 18 the EDSC. 19 Q. Then if we scroll down to the impact statement: 20 "SSC users affected have more access than is 21 required to database resources. This is contrary to 22 security policy. 23 "... There is currently no 'cost' to this issue." 24 As for "Perceived Impact": 25 "... The customer is not aware of this problem or 74 1 change. 2 "Scope: No actual impact/incidents of problems 3 relating to this issue have been experienced yet (and 4 not expected)." 5 Then if we can go down, please, to what Mr Wright 6 wrote when opening the PEAK "Summary", which we have 7 seen above: 8 "Database users do not have correct permissions." 9 Then in more detail -- and we're dealing with 10 Horizon Online here, aren't we? 11 A. Yes, we are. 12 Q. "Development have delivered scripts to allow SSC users 13 to perform certain tidyup tasks (like clear failed 14 recoveries). However they have been delivered to work 15 against an SSC role which SSC users have not been 16 granted as SSC users have the APPSUP role." 17 Can you explain what that first paragraph means, 18 please? 19 A. So these are roles in the database that grant different 20 permissions. So the SSC role is a read only role, so 21 that's our default role. The APPSUP role is the one we 22 were talking about before which does have the update 23 permissions. 24 Q. "Either SSC user creation/configuration needs to be 25 amended to make sure we have ALL required permissions 75 1 of ..." 2 Then I think that's meant to be "or": 3 "... [or] the scripts will need amending to match 4 how our users are set up in live." 5 A. Yes. 6 Q. Again, can you decode that for us, please? 7 A. So the scripts are obviously using a different 8 permission that does no longer work and either the SSC 9 profile user on the database has to be updated or the 10 scripts have to be updated, so they work. 11 Q. Then if we scroll down, please, he, that's Mr Wright, 12 I think, includes an email chain that's included. If we 13 scroll down a little further -- thank you -- I think we 14 can see an email from Anne Chambers of 1 February 2011 15 that's been cut into this PEAK. Can you see that? 16 A. Yes. 17 Q. She says: 18 "Unfortunately development write their scripts 19 explicitly to use SSC. So I think we're stuck with it 20 unless they deliver new scripts (which would not be 21 a popular or quick option). 22 "When we go off piste we use appsup. Can we have 23 both??" 24 Firstly, can you help explain what the first 25 paragraph of Ms Chambers' email is referring to? 76 1 A. So I think that's talking about the scripts that Mark 2 was detailing above, like the failed recovery tidy 3 script, that there you write them to use the SSC 4 profile, which now no longer has write permission. 5 Q. Then she says: 6 "When we go off piste we use appsup." 7 What does that mean? 8 A. So, like we were just talking about the script, that 9 script is written to -- it's a known issue about 10 clearing a failed recovery once they have been 11 investigated. "Off piste", she is basically saying that 12 there is no tool to do this, this is something we have 13 not come across before, therefore you could wait and 14 write a tool to do the correction, or we have to go in 15 manually to do the correction. 16 Q. And we use APPSUP to do that? 17 A. APPSUP is the write role, the role with the update 18 permissions. 19 Q. What do you understand the reference to going 20 "off piste" to mean? 21 A. Where there is a new issue that you haven't got a script 22 to fix already. 23 Q. Mr Gibson replies: 24 "I suspect you can have both but either way you need 25 a development fix as they produce the user creation 77 1 script which does the database bit. If they have to 2 produce a fix, I'd advise making one of the roles 3 suitable rather than having a mix of grants across both 4 roles." 5 Then scroll up, please. Mr Wright replies: 6 "I thought the original issue was why have the SSC 7 users not had the SSC role granted? If it is a bug in 8 the creation scripts then yes, needs [development] to 9 fix but I thought something was said the other day about 10 the SSC users not being set up correctly at the start?" 11 What is he referring to there? 12 A. So I think this is about the SSC users not having the 13 permissions to switch to the database roles, so that 14 they couldn't run -- the script should automatically 15 switch to whatever role it needs to do in the script and 16 it wasn't. Then he is saying "Are the SSC users set up 17 correctly? Are the permissions correct for the SSC 18 user?" 19 Q. Then if we go forward a page to your contribution. 20 Scroll down, please. Six months on, you say: 21 "This is getting confused, this incident is about 22 the SSC role which ISD ..." 23 "ISD" being? 24 A. They are the operations people. 25 Q. "... need to give to the SSC in order to run a script 78 1 provided to the SSC by development." 2 Then underneath that it seems you transferred the 3 call to a different team; is that right? 4 A. Yes, there's the host -- "APOP-Host-Dev", so APOP is 5 a database development team. 6 Q. Why was it necessary to transfer? 7 A. I think it was because I really needed an answer about 8 the database roles and what they should be set as. 9 Q. I'm not going to carry on through the PEAK, save to go 10 to the last page, please. We see Mr Haywood. We're 11 sort of a year and three months on from the start; who 12 was Mr Haywood? 13 A. The security manager. 14 Q. "The Business Impact has been updated: 15 "SSC users affected have more access than is 16 required to database resources. This is contrary to 17 security policy." 18 Then we see him including there the impact statement 19 that we read originally. Can you remember what the 20 solution was to this? 21 A. This is, I mentioned before, where we don't have any 22 default access to write permissions. I think this is 23 the outcome from this, so we have to ask SecOps to ask 24 ISD, the operations people to grant that permission for 25 a temporary process, while we do the off piste things. 79 1 So I think that was the output of this. 2 Q. When was that solution put into your memory? 3 A. After this. 4 Q. So some time after June 2015? 5 A. Yes. 6 Q. So does it follow that, between the rollout of Horizon 7 Online in, say, 2010 until mid-2015, there was off-piste 8 access by the SSC? 9 A. There was. It still wasn't the default role because the 10 default role is read only, but you could -- without 11 going through SecOps and ISD -- do set role APPSUP to be 12 granted the update permission. 13 Q. How frequently was that done? 14 A. Not very frequently, to my knowledge, but again you 15 could go through the PinICLs and PEAKs to find out at 16 that time. Sorry, OCPs and OCRs, as well, would have 17 been ... 18 Q. Was it, other than by looking at PEAKs where somebody 19 had recorded that they had done this, auditable? 20 A. I believe so. I believe there was -- 21 Q. How was it auditable? 22 A. Again, I didn't support audit but I believe that it 23 wrote a message saying that you had switched role. 24 Q. So you believe that you personally wrote a message? 25 A. No, no, sorry, the system. 80 1 Q. The system wrote a message? 2 A. The system writes a message to Audit saying that this 3 user has switched role to APPSUP. I believe, again, 4 that I think I saw a list of that in the GLO. 5 Q. Was that via a Tweet or -- 6 A. No, no. 7 Q. -- or actually seeing the evidence? 8 A. I think I saw the evidence of a list of the times that 9 they switched into it. 10 Q. Was it known within the SSC community that this going 11 off piste using APPSUP was problematic? 12 A. We didn't know it was against any rules that Mr Haywood 13 knew but going off piste, as it was put, would 14 definitely require an OCR or OCP to be raised and signed 15 off by SSC manager for OCRs and others for OCPs. 16 Q. That requires the person that's going off piste to tell 17 somebody else that they're doing it? 18 A. Yes. 19 Q. It puts the onus on the individual? 20 A. Yes. There were procedures in place and Mik was very 21 sure about his procedures and we had two sets of eyes 22 procedures as well for doing such things. 23 Q. If that was the case, that there were procedures in 24 place that included two sets of eyes on it, do you know 25 why a change was necessary? 81 1 A. I would say to make doubly sure that we couldn't do it. 2 It's another step -- there is an idea of six steps of 3 separation, where you could -- like another team can't 4 do certain things, we can't access audit, we can't 5 access the KMA, and that's a security put in and this is 6 another one of those. 7 Q. Again, in your May 2014 email, why would you not tell 8 those that were asking about this -- 9 A. I was literally asked two questions and I literally 10 replied to those two questions. 11 Q. So if you had been asked the question "Look, we're 12 looking at the extent to which the SSC can do things to 13 data without there being a proper security control 14 mechanism in place or an automatically generated audit 15 trail of them, can you tell us about any of those 16 things, please?" you may have mentioned what we're 17 talking about now? 18 A. And I would probably refer them to the audit architect 19 because we don't support audit, so I couldn't really 20 tell you that much about what does get written to audit, 21 where it gets written. 22 Q. No, but what you could say is that "We have spent, by 23 then, four years going off piste" -- 24 A. I could say that for four years we have had the access 25 to switch role to APPSUP and these are the -- probably 82 1 the times we have done it, based on the PEAKs and 2 OCPs/OCRs. 3 Q. Of course, when you were making your contribution to 4 this chain, that was in August 2011 -- 5 A. Yes. 6 Q. -- to this PEAK? 7 A. Yes. 8 Q. Did you then drop out of the PEAK thereafter? 9 A. I think I rooted it off to a different team at that 10 stage. 11 Q. So you weren't aware of, necessarily, what happened in 12 the administration of the PEAK thereafter? 13 A. Not particularly. I would have known that there was 14 a procedural change when it was changed and this is the 15 new process we got to follow to get access to APPSUP. 16 Q. But, back to the May 2014 email, it was the narrowness 17 of the questions that you were asked that caused the 18 narrowness of the answer? 19 A. I was only asked two questions so it was exactly that. 20 Q. Can we turn, lastly, to some EPOSS faults, please. Can 21 we look, please, at FUJ00036863. I think you raised 22 this PinICL? 23 A. Yes. 24 Q. Is that right? 25 A. That's correct. 83 1 Q. Would that have originated from a subpostmaster call? 2 A. No. 3 Q. Where would it have originated from? Where did it 4 originate from? 5 A. It originated inside the SSC. 6 Q. And how? 7 A. I don't know how I found that there were null modes in 8 APS and EPS transactions -- sorry, EPOSS transactions 9 but that is the key to -- 10 Q. How did you know to connect the problem with EPOSS? 11 A. So they -- we're talking about different transaction 12 types. APS transactions go into the APS database. They 13 are a type of transaction, like Bill Payments, that's 14 a APS transaction. EPOSS transactions are a different 15 type, like transacting the stamp or -- for example, yes. 16 So they are two different types of transactions and 17 where they go. 18 Q. Can we look, please, at FUJ00058190, and can we look at 19 page 8 of this document, please. 20 I think that's a rogue reference. FUJ00058190. 21 Yes, it's my fault. 22 I will ask the questions without the document 23 reference. 24 A. Sure. 25 Q. The EPOSS fault that you raised, were you aware at that 84 1 time that there was a serious instability issue with 2 EPOSS? 3 A. Only from what the PEAKs we were getting in, I would 4 say. What instability in particular? 5 Q. Were you aware that it was proposed that there should be 6 a rewrite of the code or at least the code as far as it 7 related to the cash account? 8 A. No, I wasn't aware at that time. 9 Q. Do you remember any discussions within Fujitsu about the 10 need to rewrite the EPOSS code as far as it related to 11 the cash account? 12 A. No, I wasn't aware. 13 MR BEER: Yes, thank you very much, Mr Simpkins. They are 14 the only questions I ask for the moment. 15 I believe Mr Stein is shaking his head. 16 (Pause) 17 Sir, I wonder whether we might break for a couple of 18 minutes. Ms Page wanted to raise an issue with me 19 and -- 20 SIR WYN WILLIAMS: Yes, by all means. I will stay close by, 21 so just alert me and I will come back on screen, okay? 22 MR BEER: Yes, thank you. 23 (12.17 pm) 24 (Short Break) 25 (12.23 pm) 85 1 MR BEER: Sir, can you see and hear me? 2 SIR WYN WILLIAMS: Yes. 3 MR BEER: Thank you very much. Mr Simpkins is just being 4 shown back into the room. 5 SIR WYN WILLIAMS: Sorry, would you repeat that? 6 MR BEER: Yes. Mr Simpkins is just being shown back into 7 the room. He has taken his seat now and we're ready to 8 go with Ms Page first. Thank you. 9 Questioned by MS PAGE 10 MS PAGE: Mr Simpkins, hello. I'm Flora Page. I represent 11 a number of the subpostmasters and, indeed, some of them 12 were prosecuted, as you probably know, and some of them 13 were sent to prison. So what I'm going to ask about is 14 a few different areas of how your role might have 15 affected them. 16 I'm going to start, if I may, with the third 17 supplemental agreement. Now, that may not mean much to 18 you. Have you heard of that? 19 A. I think I may have had a supplemental agreement in here 20 but I think it may have been the fourth, I'm not sure. 21 Q. So it was, just to give you a little context of 22 chronology, it was signed in January 2000, so relatively 23 early in the national rollout. You were working then, 24 weren't you, in the SSC? 25 A. Yes. 86 1 Q. One of the issues that is clear from that third 2 supplemental agreement is that the technical people in 3 Fujitsu, and indeed as a result of that agreement it is 4 clear that Post Office also knew, that there would be 5 cash account errors caused by reference data, also 6 caused by other technical faults and that, in some 7 cases, they anticipated that they would only be picked 8 up by subpostmasters phoning the call centre. Is that 9 something that you can sort of accept from me, in terms 10 of the interpretation of the agreement? 11 A. I can accept that, yes. 12 Q. All right. Well, were you and your team ever alerted to 13 that? 14 A. If the -- we would take the calls -- sorry, so they 15 would contact either MBSC or HSH and then, if it was HSH 16 it would, if it was a software issue, hopefully find its 17 way to us and then we would investigate them based on 18 that, but I don't know about the agreement. 19 Q. Well, obviously, you would be alerted if a subpostmaster 20 came to you -- 21 A. Yes. 22 Q. -- through the lower lines of support, and you would 23 know that you were speaking to a subpostmaster, but my 24 question was: did anyone at Fujitsu, in your management 25 structure or in any fashion, let you know and your team 87 1 know that there would be or there could be faults, which 2 would only become apparent because a subpostmaster 3 alerted the helpdesk to that and that might come to you 4 up through the chain? 5 A. Not particularly. I can't recall being told that there 6 would be faults that only a subpostmaster may notice, 7 but we did identify faults based off calls from 8 subpostmasters. So it was definitely a thing we did and 9 we did identify faults based on those calls. 10 If we identified a fault, we would scope the fault 11 and, once it was recognised -- and identify who was 12 affected by that, so I think I'm saying the team knew 13 that there were issues that subpostmasters were 14 identifying that weren't being picked up by automated 15 things in the data centre. 16 Q. All right. Well, in that case, can we please look at 17 document number POL00028743. When it comes up, you will 18 see that it's a PEAK from 2001. It is sometimes quite 19 hard to read these PEAKs. If we perhaps -- can you read 20 it? Are you able to? 21 A. I can read it, yes. I think this was in my pack as 22 well. 23 Q. It will have been. If we look in closely at 12.58 on 24 14 April, it says the "pm" -- I presume meaning 25 postmaster: 88 1 "... extremely unhappy about the problems with his 2 counters. He says he has had to pay out over £1,500 in 3 losses that are due to these problems. He has informed 4 POCL they can suspend him because he is refusing to make 5 good any further losses." 6 He asks for a face-to-face meeting: 7 "[He] feels very strongly about this and says he is 8 willing to take POCL to a tribunal/court because of the 9 stress he has suffered because of the problems." 10 Then it says, a bit further down, in capitals: 11 "This call is only to be closed with the express 12 permission of Julian Hall." 13 Do you know who Julian Hall is? 14 A. I don't. This was entered from the Horizon System 15 Helpdesk. This is their text before it gets to SSC. 16 Q. I see. If we go on a bit further, if we go as far as 17 page 4, please, and about halfway down we can see: 18 "This is an update for yesterday's call [this is in 19 capitals] made by the pm ... PowerHelp server was 20 down ... 21 "Call was taken over by STSA Donna Moulds and the 22 following information was manually logged: 23 "PM would like to add to the current complaint that 24 transactions are currently appearing and disappearing on 25 screen and also the PM's counter printer has not been 89 1 working either. 2 "PM had a message on screen stating [about the] 3 transaction then the screen froze and timed out. When 4 logged back in, the transaction was not on screen. PM 5 rebooted the printer, and a receipt for this transaction 6 was printed. Now the printer won't print any receipts", 7 et cetera. 8 A bit further down, it says at 9.33: 9 "PM would like to add that on the 18th April ... the 10 PM spoke to Garreth from the Environmental Team. 11 Garreth advised the PM that he will be in touch with him 12 before the end of the month to investigate any problems. 13 It is now past the end of the month, and still nothing 14 has been done." 15 If we carry on down a bit, please. This is at 9.35: 16 "PM feels the system is unreliable. PM cannot trust 17 this system." 18 He says again that he wants to speak to someone 19 face-to-face. It is quite clear, as far as this 20 postmaster is concerned, that he is saying that this is 21 not his fault, he has not done anything wrong, the 22 system is unreliable, yes? 23 A. Yes, this was a phantom transactions call, wasn't it? 24 Q. It was, that's quite right and, indeed, if we go down to 25 page 10, we can see that reference to phantom 90 1 transactions. I think a little higher -- 2 Well, while we're here we can see that it is closed 3 down on the basis that: 4 "I am therefore closing this call as [it is] 5 no fault in product." 6 A bit higher up we can see, under 12 November 2001 7 Patrick Carroll: 8 "Phantom [transactions] have not been proven in 9 circumstances which preclude user error. In all cases 10 where these have occurred a user error related cause can 11 be attributed to the phenomenon. I am therefore closing 12 this call as no fault in product." 13 But if we look further up and, in fact -- I mean, 14 you may be able to confirm it for us without us looking 15 further up, the phantom transactions that the user is 16 referring to were, in fact, witnessed, weren't they -- 17 A. Yes, by the Romec engineer. 18 Q. -- by a Romec engineer, exactly. Yet, this later entry 19 says "Well, we will just close this down, there's 20 no fault, it must be user error". 21 A. Yes, I did read through it. I don't remember Pat 22 Carroll researching this one. I know he did do a lot of 23 monitoring and things like that, that's all in the call, 24 and I don't know if this comment is after -- for after 25 those -- those were put in place but, yes, I agree it 91 1 doesn't read well. But I can't comment on what was the 2 conclusion. 3 Q. What I'm getting at here is, if you had known, if you 4 had been told explicitly and clearly that there would be 5 errors which could only be picked up by subpostmasters 6 making calls and saying that they are experiencing, 7 let's say, phantom transactions, or whatever it may be, 8 do you think you and your team would have been as 9 willing to close down calls on the basis that it must be 10 user error? 11 A. I don't know how many calls we closed down as user error 12 without good proof. Again, that probably can be 13 researched through the PinICLs and PEAKs. And this one 14 was investigated extremely heavily with multiple changes 15 made, monitoring put in, but I cannot -- I agree, 16 I cannot comment on the closure of that. 17 Q. Well, when you say you can't comment on it, what do you 18 mean by that? 19 A. I don't know what investigations Pat had concluded to 20 make that decision. 21 Q. Was there a tendency to ascribe user error if a fault 22 could not be got to the bottom of, as it were? 23 A. I have heard that mentioned before by Mr Roll, I think, 24 and I would hope not. I don't think there was. Again, 25 a retrospective review of the PEAKs and PinICLs might be 92 1 able to clarify that. 2 Q. Thank you. Could we perhaps look at another PinICL, or 3 a PinICL rather than a PEAK. This one is FUJ00042388. 4 This one begins on 25 February 2000. If we go down, 5 please, to 1 March 2000, and if we look at 11.51, we see 6 here, don't we, that at 11.51 -- Steve Warwick, he is 7 one of your colleagues, is that right? You are there at 8 the top. 9 A. He was a fourth line support. 10 Q. He was a? 11 A. Fourth line support. 12 Q. Fourth line support, I see. So does that suggest that 13 you and your colleagues have then brought him in? 14 A. Yes, so if you look at the fourth line it says "Please 15 route to EPOSS DEV". 16 Q. Right, and so he is EPOSS DEV? 17 A. Yes. 18 Q. He says, at 11.51: 19 "This is identical to an issue which was raised 20 approximately four months ago, the cause of which was 21 never found." 22 Do you know what happened when a cause was never 23 found, as it were? Who was informed? Were you ever 24 informed? Was your team ever given a message from 25 fourth line support that said "There's been no solution 93 1 to this one, it's outstanding"? 2 A. I don't know if we had that for this one, but we 3 definitely were -- we raised KELs which had 4 a description of the problem and what we have looked at 5 and they were used in order -- in case it was raised 6 again. 7 I think that there was another call later on, which 8 he said a similar issue was caused by archiving and 9 Riposte happening at the same time as doing the cash 10 account. 11 Q. If we go down to page 13, it comes back to you. Can you 12 explain to us how it comes back to you? 13 A. So Martin's routed it back to the EDSC and Diane has 14 passed it to me and I have passed it to the management 15 support unit, so I think it was raised by the management 16 support unit from the automated host detection, so 17 the -- 18 Q. Perhaps if you can just explain, so does that mean -- 19 when you say the "automated host detection", what's 20 that? 21 A. So on the TPS database, it automatically checked things 22 like cash accounts, and this was picked up -- this 23 PinICL was raised on the back of some of those alerts. 24 Q. I see. 25 A. So we are passing it back, the information on the 94 1 PinICL, back to the team who raised the call at that 2 point. 3 Q. I see, and so when it says, a bit further down, "POCL 4 have now agreed closure of this incident", that's 5 because this is something that's arisen on a platform 6 and therefore the customer support people are actually 7 liaising with POCL about it? 8 A. Yes. 9 Q. And POCL have agreed to close this down? 10 A. Yes, so the MSU, the -- or BSU at that time, they 11 would -- for -- send corrected cash accounts to POCL, so 12 this is what their process was, and then -- 13 Q. So this would definitely have involved amending cash 14 accounts? 15 A. It was involved in reporting the corrected cash 16 accounts, not touching the system at all in any way, but 17 reporting -- 18 Q. But explaining that there was -- 19 A. Explaining what the -- why the cash accounts that would 20 have been sent automatically were incorrect. 21 Q. When you say "explaining" -- we can look at it if you 22 like -- but from your memory, having read it, it does 23 make it pretty clear, doesn't it, that the problem is 24 pretty intractable. This doesn't appear to have 25 resolved the problem, does it -- 95 1 A. Correct. 2 Q. -- on a root cause basis? 3 A. Yes. 4 Q. Indeed, it's obviously involving Riposte, it's involving 5 the DataServer, it's a pretty deep problem, if I can put 6 it that way? 7 A. Yes. 8 Q. And this record does not show it having been resolved? 9 A. Correct. 10 Q. But, at the end of it, through the customer support team 11 liaising with POCL, they have at least resolved the cash 12 accounts, if not the problem? 13 A. Correct. 14 Q. You, presumably, have no idea what was then decided in 15 terms of how cash accounts were going to be looked at or 16 handled or dealt with going forward? 17 A. No. I believe the management support unit would send 18 a corrected cash account in these instances -- 19 Q. For the ones that had been found? 20 A. For the -- yes, and so this was picked up on the 21 automated checks. 22 Q. Yes, I see. It is fair to say, isn't it, that cash 23 account discrepancies came up a lot in what you were 24 dealing with, didn't they, at that time? 25 A. They did. 96 1 Q. Was there any forum for collating those and putting them 2 together and saying "Here's a lot of cash account 3 problems, can we spot any patterns here"? 4 A. One of the documents I had was -- had all the issues 5 that had been fixed, or listed them all as -- and which 6 were new ones that affected the cash accounts. 7 I presume it was something to do with the AI -- I can 8 have a flick through, but it had a table at the back, 9 and it seemed to indicate all the ones that -- and how 10 they were being detected. 11 Q. But that wasn't your document? 12 A. No. 13 Q. That wasn't a document produced by SSC? No, all right. 14 MS PAGE: Thank you, those are my questions. 15 SIR WYN WILLIAMS: Anyone else? 16 MS PATRICK: Yes, sir, Ms Patrick here for Hudgells' CPs. 17 SIR WYN WILLIAMS: Yes. 18 Questioned by MS PATRICK 19 MS PATRICK: Good morning, Mr Simpkins. My name is Angela 20 Patrick and I ask questions for another number of 21 subpostmasters who were wrongly convicted and I'm 22 instructed by Hudgells solicitors. You will be glad to 23 know I only have two topics to ask you about and it's 24 about issues that arose in the management of bugs, 25 errors and defects and the first document I would like 97 1 us to take a look at is FUJ00081584. 2 You see there is a table at the top there and it 3 looks like it is a note of a meeting and I think you can 4 see there you can an attendee, there recorded. 5 A. Yes. 6 Q. Your name is a few from the bottom and right below yours 7 is Gareth Jenkins. Can you see that? 8 A. I can. 9 Q. We think this is a table -- you can see at the top, it 10 is about a receipts and payments mismatch issue and the 11 Inquiry has heard something about that and will hear 12 something more. I think that was issue that was 13 discovered in mid-2010; is that correct? 14 A. Yes, newly into the HNG-X. 15 Q. Yes, newly into the development of Horizon Online; is 16 that fair? 17 A. Yes, that's fair. 18 Q. The only reason I raise that is because there's no date 19 on the document. 20 A. No, that's fair. 21 Q. If we can go to page 3, please, at the very top of the 22 page and we can see there there's an explanation about: 23 "The receipts and payment mismatch will result in 24 an error code being generated which will allow Fujitsu 25 to ..." 98 1 There's a bit more explanation but what I want to 2 look at, at the bottom, is that paragraph: 3 "We have asked Fujitsu why it has taken so long to 4 react to and escalate an issue which began in May. They 5 will provide feedback in due course." 6 So you said the bug was discovered in the period 7 running up to the development of Horizon Online. Was 8 this actually in the period which was running up to the 9 acceptance of Horizon Online? 10 A. I don't know. 11 Q. Okay, we can perhaps ask another witness. 12 Do you know why there was a delay in informing the 13 Post Office about this bug? 14 A. No, I don't know. 15 Q. Are you able to help us on where the feedback that's 16 mentioned there, that was going to be provided to the 17 Post Office, could be found? 18 A. Not to my knowledge, unless it was the list of the 19 affected branches. I believe that there was a list 20 produced and monitored for further occurrences but 21 I couldn't -- 22 Q. I think my reading of that is Fujitsu -- 23 "We have asked Fujitsu why it has taken so long to 24 react to this and escalate an issue which began in May. 25 They will provide feedback in due course." 99 1 Do you know if there was any feedback given to the 2 Post Office about why there was such a long delay in 3 informing them about the bug? 4 A. No, I don't. 5 Q. Thank you. This receipts and payments mismatch bug, are 6 you able to help us with your explanation, perhaps 7 a simple explanation, of what it was? 8 A. So I have read a little bit up on it, so it was when you 9 were doing your stock unit balance and, if you had 10 a discrepancy, it comes up with a message to warn you 11 and say whether you want to post it to a local suspense. 12 If you, at that point, hit the "Cancel" on the message 13 you could then hit "Print" and carry on forward. It 14 doesn't rewarn you and it lost that discrepancy value. 15 So it produces a cash account -- I'm sorry, a stock 16 unit rollover that was out of balance so the payments 17 didn't match receipts. It was visible on the payroll, 18 but it didn't warn the postmaster again. 19 Then if they went to do the branch trading 20 statement, when they roll the branch trading statement, 21 they would get a non-zero trading position warning 22 message because that stock unit had a payments/receipts 23 mismatch. 24 Q. So, really short, it showed an imbalance in the cash 25 account ultimately, didn't it? 100 1 A. It showed an imbalance in the branch trading statement, 2 yes. 3 Q. Thank you. Now, if we can turn to page 2, please, we 4 can see at the bottom of page 2 where the impact of this 5 was analysed. And I'm going to look at the last three 6 of those bullets, "Impact": 7 "If widely known [this] could cause a loss of 8 confidence in the Horizon System by branches. 9 "Potential impact upon ongoing legal cases where 10 branches are disputing the integrity of Horizon data." 11 Then, finally: 12 "It could provide branches ammunition to blame 13 Horizon for future discrepancies." 14 You can see that there on the record. 15 A. I can. 16 Q. So that's discussing that the impact wasn't simply on 17 the inability of subpostmasters to reach a balance but 18 there could be a wider impact because of the 19 understanding of this problem being a system problem; is 20 that fair? 21 A. I think that's fair. 22 Q. Can we look at page 3, please, at the bottom. We have, 23 I think, here -- if I'm correct in my pagination -- 24 a list of possible solutions; is that right? 25 A. Yes. 101 1 Q. There's 1, 2 and 3, and if we look at solution 2 there's 2 a number of suggestions there: 3 "P and BA will journal values from the discrepancy 4 account into the customer account and recover/refund via 5 normal processes. This will need to be supported by 6 an approved POL communication. Unlike the branch 7 'POLSAP' remains in balance, albeit with an account 8 discrepancy that should be cleared." 9 I think that the recommendation you can see there is 10 that that solution, solution 2, should be progressed; is 11 that right? At the top, under "Proposal for affected 12 branches". 13 A. Yes, the group's recommendation is that solution 2 14 should be progressed. 15 Q. Are you able to help us as to what happened: was 16 solution 2 adopted? 17 A. I'm not able to help you. I'm sure it should be 18 relatively straightforward to find out. 19 Q. Can we scroll to the top of that page, page 3. You can 20 see there's an introduction and a sort of overview 21 explained there. At paragraph 2: 22 "Fujitsu are writing a code fix which ..." 23 I think there is a "will" missing there: 24 "... which [will] stop the discrepancy disappearing 25 from Horizon in the future. They are aiming to deliver 102 1 this into test week commencing 4th October. With live 2 proving at the model office week commencing 3 11th October, with rollout to the network completed by 4 21st October. We have explored moving this forward and 5 this is the earliest it can be released into live." 6 So the problem was discovered in May, it's brought 7 to the attention of Post Office, I think, in September, 8 and now the solution will not be actioned or live until 9 October; is that correct? 10 A. That sounds correct, yes. 11 Q. Then it goes on: 12 "The code fix will ..." 13 I think there's another typo here: 14 "The code fix will on stop the issue occurring in 15 the future but it will not fix any current mismatch at 16 branch." 17 Can you help us with what that would mean in 18 practice? 19 A. So if you have already got a payments/receipts mismatch 20 in the stock unit and a non-zero branch trading 21 statement, this fix won't correct that, but it will stop 22 it happening in future. 23 Q. So there would be a problem that wouldn't be fixed by 24 the fix. Does that mean that something else would need 25 to be done to address -- 103 1 A. The solution -- 2 Q. Apologies. Would something else need to be done to fix 3 the mismatch that had already happened? 4 A. You don't have to do anything, apart from you -- the 5 imbalance would rollover -- be brought forward and then 6 be reported in the next branch trading statement and 7 after that it would be cleared. So you don't actually 8 have to do anything as long as the Post Office is made 9 aware of what has happened. 10 Q. As long as the Post Office is aware that it has 11 happened -- 12 A. Correct. 13 Q. -- and they are aware which branches may have been 14 affected? 15 A. Correct. 16 Q. Thank you. I think we can move to the second document 17 I would like to look at, Mr Simpkins. It's FUJ00083770. 18 It's a series of emails. Can you see that in front of 19 you now? 20 A. Yes. 21 Q. You can see, if we scroll to the very bottom, which I'm 22 going to start with, where we see your name mentioned 23 first -- you can see there at the very bottom there was 24 an email sent from Mike Stewart to you on 25 22 February 2006. Can you see that? 104 1 A. Yes. 2 Q. I'm not going to look at that yet, I'm going to scroll 3 a little bit to the bottom, so that we can all see what 4 the issue was. Can we go to page 6, please, at the 5 bottom, a little further down, thank you. You will see 6 there's an email there from Shaun -- it's Shaun -- it's 7 from Sandra McKay to Shaun Turner. Are you able to help 8 us with who they were? 9 A. No. 10 Q. I think her title -- somebody else might be able to help 11 us. Sandra McKay, it says, is from sales and service, 12 and it says: 13 "You may recall that in September the above office 14 had major problems with their Horizon System relating to 15 transfers between stock units. 16 "The subpostmaster has reported that he is again 17 experiencing problems with transfers ... which resulted 18 in a loss of around £43,000 which has subsequently 19 rectified itself. I know that the subpostmaster has 20 reported this to Horizon support, who have come back to 21 him stating that they cannot find any problem. 22 "Clearly the subpostmaster is concerned as we have 23 just spent a number of months trying to sort out the 24 first instance and he doesn't want a repeat performance. 25 He is convinced that there is something wrong with his 105 1 Horizon kit. I would be grateful if you could 2 investigate this and give him any support that you can." 3 If we scroll back up a little bit, we will see the 4 reply, or a further email in the chain, at the top of 5 page 6. I think it's a further email in the chain 6 rather than a reply. It's from Brian Trotter to Shaun 7 Turner and do you know who Brian Trotter was? 8 A. No. 9 Q. So it says: 10 "Further to Sandra's email, I visited the branch 11 with Sandra last week and the subpostmaster provided 12 clear documented evidence that something very wrong is 13 occurring with some of the processors when carrying out 14 transfers between stock units. To be absolutely sure 15 from our side, can we either carry out a thorough check 16 of the alleged faulty processors or swap them out." 17 So from what we can see at this end of the problem, 18 it's the postmaster who has had an issue which has come 19 back again and is being investigated, and somebody has 20 also again witnessed that there is indeed an issue; is 21 that fair? 22 A. That's fair. 23 Q. Okay. If we can go back to page 1, please, and if we 24 start at the bottom, with the email that Mike Stewart 25 sent to you. If we scroll a little bit further down, 106 1 please, we see there Mike Stewart is writing to you: 2 "John, did you get a chance to look at this? Do we 3 think all will be well after S90 counter rollout?" 4 Was the S90 a new release -- 5 A. Yes. 6 Q. -- of Horizon Legacy? 7 A. Yes. 8 Q. If we go down, we can see -- I won't read it all out -- 9 in the first instance, he has tried to reach Anne 10 Chambers but she was away; is that right? 11 A. I presume so. 12 Q. You see it says, "Anne is away, could I have your 13 comments as you were involved as well". He goes on to 14 talk about the PinICL and he refers to there being 15 a PinICL for this issue, and it says: 16 "The time out events are apparently fixed in a new 17 Riposte version released at S90." 18 There's a PinICL number, and it says: 19 "I have looked at the problems and can't see why the 20 system reported disconnected nodes." 21 He goes on a little bit, and he explains: 22 "I think the best thing now is to see what happens 23 after S90. I'll continue to keep this call open to 24 remind me that this site should be checked then." 25 It goes on a little bit to talk about the 107 1 postmaster, or the person reporting the problem 2 initially being reported as female, and it goes on again 3 to say a little bit more about the problem. He refers 4 to "a magical £43,000 appearing and disappearing" and 5 the postmaster is then reported to be male, and he says, 6 September: 7 "... the above office had major problems with their 8 Horizon System relating to transfers between stock 9 units. 10 "The subpostmaster has reported that he is again 11 experiencing problems with transfers ..." 12 It says: 13 "... which resulted in ... around £43,000 which has 14 subsequently rectified itself. I know that [he] has 15 reported this to Horizon support who have come back to 16 him stating that they cannot find any problem." 17 It repeats almost the content of the email we have 18 just discussed. 19 He goes on, he says: 20 "Sorry for this long-windedness. Is it a problem at 21 the branch he wants to query? Is it Horizon kit or is 22 there an issue with staff there? 23 "If there is an issue is this S90 release the cure? 24 How confident are you/we it will fix the problem?" 25 Then he says the release is due in the week of 108 1 4 March. So we're in February at this point, he is 2 talking about a few weeks away; is that fair? 3 A. Yes. 4 Q. So he is posing some questions for you, originally for 5 Anne Chambers to consider; is that right? 6 A. Yes. 7 Q. If we scroll up on page 1 we will see the reply doesn't 8 come from you, it comes from Anne Chambers and it says: 9 "I believe John has already responded to this, so 10 don't know if you need any more from me ..." 11 It perhaps suggests that she has spoken to you 12 before she has replied, doesn't it? 13 A. That seems reasonable. 14 Q. It says: 15 "I haven't looked at the recent evidence but I know 16 in the past this site had hit this Riposte lock problem 17 2 or 3 times within a few weeks. This problem has been 18 around for years and affects a number of sites most 19 weeks, and finally Escher say they have done something 20 about it. I am interested in whether they really have 21 fixed it which is why I left the call open -- to remind 22 me to check over the whole estate once S90 is live -- 23 call me cynical but I do not just accept a third party's 24 word that they have fixed something! 25 "What I never got to the bottom of ..." 109 1 She explains she is concerned why this particular 2 branch had a particular problem. It goes on to say: 3 "... KELs tell SMC that they must contact sites and 4 warn them of balancing problems if they notice the event 5 storms caused by the held lock and advise them to 6 reboot ... before continuing with the balance." 7 It says: 8 "Unfortunately in practice it seems to take SMC 9 several hours to notice these storms by which time the 10 damage may have been done." 11 So it's a problem there that we know has already 12 been known about for years; is that right? 13 A. The locking problem, yes. 14 Q. There's no solution as yet. They're looking to S90; is 15 that correct? 16 A. Yes, so the locking problem stops counters communicating 17 between each other, so it's like having -- what we were 18 talking before about replication not happening and what 19 had happened is with the (unclear) square they did 20 a transfer out, they did the transfer in on one counter 21 and then they could do the transfer in on another 22 counter because it hadn't got the transfer in messages 23 replicated to it, so they had two transfer ins and one 24 transfer out, so you had a payments and receipts 25 mismatch. 110 1 Q. So again a mismatch. 2 She says, Anne, that they still need to check the 3 whole estate after S90 goes live? 4 A. That's looking for these events. 5 Q. And there's still a need to investigate further, isn't 6 there? 7 A. Yes, so once the software has been rolled out, then you 8 would need to check to ensure it has fixed the problem. 9 Q. But she expresses the problem that: 10 "With this issue it can sometimes take several hours 11 to detect the problem and by that point the damage has 12 been done." 13 A. That was the workaround until the fix is put in place, 14 so the workaround was that SMC monitor the events from 15 the estate and the lock event -- when they see the lock 16 event, they contact the branch to reboot that counter, 17 which was the workaround to fix the locking problem, so 18 that then it will be replicating to the neighbouring 19 counters. 20 Q. If we go back to the email chain, the very last in the 21 thread, we see a message from Mike Stewart to Anne 22 Chambers which says: 23 "Anne, John did reply but just to say that Escher 24 say they have fixed it? So, like you, we will have to 25 wait and see what happens after S90 rollout." 111 1 Again, what we've got there -- there's no solution 2 as yet. Everybody is going to wait and see for S90; is 3 that fair? 4 A. That's fair. The workaround is in place, but it is 5 fair. 6 Q. Again, the issue is that Escher say they have fixed it? 7 A. Yes. 8 Q. There's no certainty at this point, is there? 9 A. No, I think the third line support team are a cynical 10 lot. 11 Q. Indeed. Were Fujitsu here reliant on Escher for 12 a solution? 13 A. A fix to Riposte. This is a Riposte bug with the 14 Riposte locking, so it wouldn't replicate. 15 Q. So for a Riposte bug -- 16 A. Yes. 17 Q. -- it needs an Escher fix? 18 A. Escher write the Riposte software, yes, that's correct. 19 Q. You weren't fixing it onsite, it had to be repaired by 20 Escher? 21 A. That's correct. It's a software -- the new version of 22 Riposte fixed that problem. 23 MS PATRICK: Thank you. I don't think I have any further 24 questions for you, Mr Simpkins. 25 MR BEER: Sir, I don't think there are any other questions 112 1 from anyone. 2 SIR WYN WILLIAMS: So that's obviously a convenient time to 3 break for lunch. 4 Mr Simpkins, thank you very much for providing your 5 written statement and for answering questions during the 6 course of the morning. I'm grateful. 7 A. Thank you. 8 MR BEER: Sir, before we break, there is a possibility that 9 Mr Simpkins will come back to help us further in the 10 Inquiry in later phases. I have been asked by the 11 Fujitsu legal team whether they have permission to speak 12 with him in the intervening period. 13 SIR WYN WILLIAMS: Well, my short answer to that is yes. 14 MR BEER: Yes. Thank you very much, sir. So did you say 15 2.00? 16 SIR WYN WILLIAMS: Well, 5 past, I think. 17 MR BEER: Did you? Thank you very much, sir. 18 (1.05 pm) 19 (The luncheon adjournment) 20 (2.04 pm) 21 MR STEVENS: Good afternoon, sir, can you see and hear me? 22 SIR WYN WILLIAMS: Yes, I can. 23 MR STEVENS: Thank you. If I may call Mr Ascott. 24 MARK ASCOTT (sworn) 25 Questioned by MR STEVENS 113 1 MR STEVENS: Mr Ascott, as you know my name, is Sam Stevens 2 and I ask questions on behalf of the Inquiry. Please 3 could I ask you to state your full name? 4 A. Mark Andrew Ascott. 5 Q. Thank you for giving evidence to the Inquiry today. 6 I want to start with the bundles in front of you. 7 You should see a witness statement, and that is dated 8 9 August 2022, and at page 24 of that statement is that 9 your signature? 10 A. Yes, it is. 11 Q. Are the contents of that statement true to the best of 12 your knowledge and belief? 13 A. To the best of my knowledge and belief, yes. 14 Q. Thank you. That now stands as your evidence to the 15 Inquiry but I will be asking you some questions, not on 16 all parts of it and, in particular, I won't be asking 17 you questions today on the elements relating to Horizon 18 Online, or at least not in any depth. 19 In your witness statement, you say that in 20 August 1998 you worked for a Fujitsu business called The 21 Solution Centre; is that right? 22 A. That's correct. 23 Q. At that stage could you summarise what qualifications 24 you had relevant to IT? 25 A. So, as a member of The Solutions Centre I was assigned 114 1 to various projects which would be involved with 2 integrating, implementing and testing solutions for 3 customers. At the end of August, I had completed 4 three years on assignment to Barclays Investment Bank, 5 where we had migrated them from 40 locations around 6 Tower Hill and Royal Mint Court, down to Canary Wharf, 7 so I had been involved in assisting a senior technical 8 design authority in implementing NT domains and 9 providing work stations and configuring work stations 10 which were rolled out to the various BZW teams for their 11 use in their new location. 12 Q. Thank you. You say you were assigned to ICL Pathway 13 from The Solution Centre between September and 14 December 1998 to work on testing. 15 A. Mm-hm, that's right. 16 Q. What specifically were you testing at that point? 17 A. I was testing network access and the ability for the 18 network to be robust to withstand sort of attacks, so 19 man in the middle type attacks, those types of spoofing 20 IP addresses, to spoof devices that shouldn't be 21 accessing -- and those types of tests. 22 Q. So you weren't testing the EPOSS application? 23 A. No. 24 Q. So, as you say, you transferred to the Pathway 25 organisation, you say, in January 1999 and you worked on 115 1 the secure builds development team, which we will 2 discuss in due course, but that was under Alan 3 D'Alvarez, was it? 4 A. That was under Alan D'Alvarez, yes. 5 Q. So it was in the security domain? 6 A. That's my belief, that Alan was looking after the 7 security implementation. 8 Q. You then say, in 2000, you moved to work for a team 9 known as infrastructure products development unit. What 10 role did that entail? 11 A. Well, initially, I continued with the secure builds, so 12 I was on the secure buildings development team. 13 I worked in the Bracknell office but I eventually was 14 moved across to work in the Feltham office reporting 15 into Pete Dreweatt and Ian Morrison. 16 Q. Thank you. We will come to deal with that team in 17 a moment. You left the Post Office account in 18 August 2005 -- 19 A. I did. 20 Q. -- and you returned, I think, in July 2008 -- 21 A. That's correct. 22 Q. -- and you went on to work on Horizon Online? 23 A. Yes, I did. 24 Q. Just one point on that. In May 2009, there was 25 an article in Computer Weekly by Rebecca Thomson, which 116 1 criticised the Legacy Horizon IT system at that stage. 2 Were you aware of that article at the time? 3 A. I don't recall that specific article. I would have been 4 aware that Computer Weekly and Computing would be 5 writing articles but I wasn't taking much notice of 6 those. 7 Q. It wasn't sort of spoken about in the office, that 8 a core product that your account was working on had been 9 criticised by a trade journal? 10 A. I think there would have been discussion within the 11 office but, from my part, I tend not to believe what 12 journalists write. For me, when I started working for 13 ICL in 1978, I read religiously Computing and Computing 14 Weekly and -- but, over time, I just believed that those 15 publications were, you know, biased against ICL, so it 16 wouldn't have been a surprise to me that there would 17 have been negative articles in those publications. 18 Q. I want to move to your witness statement. If we could 19 bring that up please, it's WITN04760100 and 20 paragraph 34, please. Thank you. 21 I apologise, I have the wrong reference there. What 22 I will do is say this: in your statement, you refer to 23 having -- you say you recall that: 24 "High level test plans and test reports included 25 Post Office staff as recipients and reviewers of draft 117 1 and approved documents." 2 Yes? 3 A. Yes. 4 Q. Was that in relation to Legacy Horizon or Horizon 5 Online? 6 A. It would have been Horizon Online. 7 Q. In respect of Legacy Horizon, would that -- that 8 statement have held true? 9 A. I'm not sure on that. I wasn't part of the testing team 10 for -- down at Feltham, that would have been testing 11 with counters, so I am aware that Post Office staff did 12 work closely with the SV&I team -- 13 Q. But you didn't have direct -- 14 A. But I didn't have direct involvement in the production 15 or necessarily the reviewing of those documents. 16 Q. Does the same apply for design and development documents 17 as well? 18 A. I would say the design documents around the security 19 solution that I would have been involved in didn't 20 generally include Post Office recipients. 21 Q. I want to then -- in your statement you, under 22 a heading, give quite a lot of detail on the testing 23 that you say you were involved with as part of the 24 Horizon IT system. Was that the testing you did for the 25 network matters you discussed earlier? 118 1 A. No, the testing that I'm referring to in my witness 2 statements are mainly based upon my experiences of 3 Horizon Online. 4 Q. Horizon Online, not Legacy Horizon? 5 A. Mm-hm. So I was aware of the testing team in Feltham 6 where the system tests, the SV&I test rig, there was 7 also a test rig that was dedicated to supporting 8 performance testing and there was also a test rig that 9 was used for release testing, so I met and worked with 10 a number of the people in the test team, in my role as 11 a developer, in supporting them to diagnose defects with 12 the products, which myself or my team had caused them to 13 have to find. 14 Q. Could I just stop there. You referred to test rigs. 15 Just in lay terms or for a non-expert, what exactly is 16 a "test rig"? 17 A. My definition of a test rig is a combination of 18 platforms and servers which are going to replicate the 19 systems which will be used in the live service and, in 20 relation to Legacy Horizon and Horizon Online, 21 a combination with counters. 22 So some rigs would not necessarily have counters, 23 you may create a -- what we would refer to as a harness, 24 where that just involves servers, such as, when I was 25 looking after FTMS, we could have a test rig that just 119 1 involved the local FTMS gateway and the remote FTMS 2 gateway -- 3 Q. Sorry, just on that, can you explain what "FTMS" is? 4 A. FTMS was the file transfer managed service application 5 used to move data from the database servers to other 6 servers that was going to process that data. 7 Q. So you have given us a definition of what the test rigs 8 are and some of your involvement in it. Did you have 9 any involvement in -- or do you have any knowledge of 10 how the process of balancing was tested? 11 A. No. I wasn't involved in the sort of counter operations 12 and the transactions and the processes during Horizon 13 Legacy system, or even Horizon Online. I did have more 14 involvement with counters during Horizon Online doing 15 performance testing. That would have involved creating 16 scripts which could be driven through a tool we used, 17 LoadRunner, to create a load and those scripts would 18 have embedded counter transactions. 19 And part of that would have involved processing of 20 data created by the counter transactions within the data 21 centre systems, primarily the database servers. 22 Q. So, in essence, what we have here is a high level 23 description of testing but your working knowledge of 24 testing for Legacy Horizon is not strong? 25 A. Yes, so in the secure builds side, where I was working, 120 1 I was generating scripts, which would translate the 2 policy files and the requirements around the security 3 needs that would result in the platforms being built in 4 the test rigs, in a manner that was going to replicate 5 the live service so ... 6 Q. Well, I do want to come to that shortly. Before moving 7 on, you refer to something in paragraph 31 of your 8 statement, "Maestro24x7 batch scheduler". Was that 9 a Legacy Horizon matter? 10 A. It was an application as part of the Legacy Horizon 11 solution, which coordinated jobs that would be run in 12 a 24-hour period in a sequence that enabled data to be 13 harvested and collected and then processed by the 14 various subsystems. 15 Q. So you have already mentioned from testing that defects 16 would be raised and they would need to be addressed by 17 development teams, and that was through a system called 18 PinICL. 19 A. It was. 20 Q. That's a repository to log these identified defects and 21 maintain a central point of what has been done to 22 investigate them and resolve them. Do you agree with 23 that? 24 A. I agree that PinICL was that system that was used for 25 defect management. 121 1 Q. We have heard evidence about individual PinICLs being 2 prioritised. Do you recall firstly who prioritised them 3 in your team? 4 A. That could be a combination of people within the team. 5 It could be the programmer that was responsible for that 6 product, it could be myself and that programmer. It 7 could be that we would seek guidance from the senior 8 designers that were available to us, just to identify 9 what the true impact would be. 10 Q. How would you go about prioritising those PinICLs? 11 A. In terms of how quick we could fix them. One of the 12 things which we learned over time was what business 13 impact they would have but, initially, we were making 14 assessments on "That's a typo, we can correct a typo 15 very quickly and easily", or, you know, "That's going to 16 be a complex piece of work to fix", and we would, 17 generally, in those situations, turn to the designers to 18 guide. 19 Q. So, sorry, for the more complex fixes, would they be 20 prioritised higher or lower? 21 A. They would likely be prioritised higher. 22 Q. And the easier ones prioritised lower, but you weren't 23 assessing that on business need, initially? 24 A. Not initially. In the secure build space, where I was 25 working, the defects that would come back to myself 122 1 would usually prevent part of a test rig from being 2 built or it may prevent a test rig from being built at 3 all. So, clearly, if it was an issue where the test rig 4 couldn't be built, that was important and that would be 5 prioritised highly. 6 If I had made a typo, a group had been named 7 incorrectly, or a user account had been specified 8 incorrectly, that would have been a lower priority 9 defect and would have been fixed in accordance to the 10 priorities that were set at the time. 11 Q. Are you aware of anyone outside of ICL Pathway who had 12 access to the PinICL database? 13 A. I'm not aware of anybody that wasn't a member of Pathway 14 having access to the PinICL system. 15 Q. So the Inquiry has heard evidence this week concerning 16 problems with EPOSS and that application, one of which 17 was related to the malformed or incomplete messages 18 created by Riposte. Were you aware of that at the time? 19 A. No, I wasn't. 20 Q. When did you become aware of issues with Riposte? 21 A. I think I would have been aware of Riposte issues around 22 about 2001/2002. 23 Q. Why was that? 24 A. Working more and more in Feltham, I had the wider 25 Pathway design and development team around me, so 123 1 I would have likely encountered people that would have 2 been involved, designers that would have been involved 3 in that counter solution. There would have been, like, 4 chit-chat in the canteen, that type of thing. 5 Q. So in around 2001/2002, is this roughly the point where 6 you started attending the morning prayers meetings? 7 A. Certainly once I had taken on responsibility for 8 managing the infrastructure teams, which I listed in my 9 statement, there was a higher chance that, you know, one 10 of those applications would require some representation 11 at morning prayers. 12 Q. Could you just -- just so we can hear what your -- why 13 was it called "morning prayers"? 14 A. Simply because it was a very early morning meeting. 15 I guess the reference goes back to, sort of, the 16 religious context of, you know, morning matins and -- 17 Q. What did you do in the meetings? 18 A. I would attend with other attendees. I would listen to 19 the issues that were being described there. Some may 20 not involve me at all, but if there was an issue that 21 was affecting the application set that I have listed in 22 my witness statement, then the likelihood is I would be 23 asked to take away and investigate what could be done to 24 resolve a PinICL as quickly as possible. 25 Q. Could you just take us through the application set in 124 1 non-expert terms, as it were, to say what you were 2 actually responsible for at these morning prayer 3 meetings? 4 A. Yes, so secure builds, which was defining the NT domain 5 structure, putting together the secure roles and secure 6 role templates, defining the access control lists, which 7 would be implemented against the NT platform set. FTMS, 8 which I have described as the file transfer manager 9 service, that application was used to move data, for 10 example the TIP data from the database servers across to 11 the Post Office systems. FTMS was also moved to move 12 data to GiroBank for onward processing from GiroBank. 13 Audit-Dev, so there was the audit and archive 14 solution which I was responsible for collecting data 15 which would be stored for a period, which I knew to be 16 seven years, and, you know, there was a set of 17 definitions as to which data should be collected and 18 recorded in that repository. Maestro-Dev, which we have 19 touched on, so the driving part of the solution that 20 kept the data flowing as and when it needed to be during 21 the 24-hour cycle. 22 The other area that I was involved in was the auto 23 configuration database, so that was the database that 24 the counters would call into when they were being built 25 and, on the counter, there was a utility called 125 1 PC Config and Europa. Europa would do checks and 2 trigger PC Config as needed for a new-build counter. 3 PC Config would connect and then, if you like, the 4 personalisation data would be provided by the 5 Auto Config database to that counter. 6 Q. So if and when there were discussions on problems with 7 Riposte and/or the EPOSS application, would you be 8 involved in those conversations other than as 9 a listener? 10 A. I wouldn't have been involved in those conversations 11 other than as a listener. 12 Q. At the morning prayer meetings, was this solely limited 13 to people employed by ICL Pathway and later Fujitsu? 14 A. Yes. All the representation there would have been ICL 15 or Fujitsu people. 16 Q. You mentioned then that one of the areas over which you 17 had responsibility was audit. Would you have been 18 responsible for -- or would that have included the audit 19 of actions taken by support services such as the SSC? 20 A. I believe so. 21 Q. Presumably, that also refers to audit for recording 22 transactions that are carried out in Post Office 23 Counters as well? 24 A. Yes, and the movement of data. So when -- for example, 25 FTMS would collect files from the database servers. 126 1 That data would have been recorded and stored in the 2 audit servers. 3 Q. Was there a record of -- an audit record of key strokes 4 that a subpostmaster would use when using a Horizon 5 terminal? 6 A. That I don't know. 7 Q. Could you tell us what the ARQ audit data contained? 8 A. No, I wouldn't be able to tell you what data is recorded 9 in it. I believe it would include a PAN number, 10 a unique number associated with a debit card or credit 11 card, but whatever data it would have recorded, I don't 12 know. 13 Q. Can I then turn -- I want to turn to the secure build 14 Windows script and the secure build -- I think "Secure 15 Builds-Dev" it is referred to in the document, is that 16 the team? 17 A. Yes, secure builds is what we were known as, the "Dev" 18 was implied. 19 Q. We have heard from Fujitsu employees this week evidence 20 stating that it would be necessary for third line 21 support to have write access to counter systems. Would 22 you agree with that? 23 A. Yes, I would. 24 Q. Subject, of course, to proper control over its use? 25 A. Yes, so the requirements for the secure roles were 127 1 recorded in the document RS/Req/012, which I think Alan 2 D'Alvarez authored, and that document there identifies 3 the secure roles that needed to be created and listed, 4 if you like, the attributes of each secure role, so ... 5 Q. Could we bring up FUJ00087994, please. Is this the 6 document you're referring to? 7 A. Yes. 8 Q. As you say, it is drafted by Alan D'Alvarez. Did you 9 have any input into it? 10 A. I think I would have had discussions with Alan to sort 11 of, if you like, shape what that group definitions would 12 look like. 13 Q. We don't need to turn it up but, in your witness 14 statement, you say, at paragraph 20: 15 "I regarded the security architects I worked with as 16 subject matter experts and deferred to their knowledge 17 and design thinking. If I did not understand part of 18 their designs, I would discuss my concerns with them so 19 that I could gain a complete understanding of their 20 designs. Once the NT domain designs were approved by 21 the senior design team in Fujitsu and were due to be 22 taken forward, I developed tooling and scripts that were 23 used to implement the secure builds on the various 24 Windows platforms." 25 A. Yes. 128 1 Q. So is this saying that you would use this document and, 2 when you say the security architects, is that referring 3 to Alan D'Alvarez? 4 A. No, I'm referring to Belinda Fairthorne there, so 5 Belinda was ICL's senior security architect. Certainly 6 in 1990, I was part of an organisation of which Belinda 7 was a key member of the senior architects group and we 8 reported into ICL's chief technology officer at the 9 time. 10 Q. She was the person who wrote the access control policy? 11 A. She -- she certainly contributed to that production of 12 that document, yes. 13 Q. When it came to designing the scripts to make, as you 14 say, the secure NT build, would this have been the 15 primary document you would have used? 16 A. This would have been one of the documents that I used. 17 I had meetings with Belinda and Barry Procter to go 18 through a set of rules and requirements which would then 19 provide the framework that would enable me to shape the 20 NT domain design. 21 So that NT domain design was the start point for the 22 work that I did with Belinda. From the NT domain design 23 document, which in this group definitions for secure NT 24 build document that Alan authored, in the tables that 25 are included in this document, the columns include the 129 1 domain names, and Alan would have needed those domain 2 names in order to fit the group definitions for the 3 secure roles to build that picture up. 4 Q. Well, let's look at one of those tables. If we could, 5 please, turn to page 9, thank you. So it says "Group 6 Name to be implemented", on the left, and "SSC Apps 7 MAN", that's SSC application management? 8 A. Yes. 9 Q. You were referring to the domains earlier. Which part 10 of this table were you referring to? 11 A. So, in this table here, the authentication domain column 12 lists the authentication domains where users accessing 13 those domains are going to be verified in terms of user 14 and passwords. 15 Q. So could you assist us in decoding what's under the 16 "Authentication Domain" column? 17 A. So the "PWYDCS", that domain name refers to Pathway data 18 centre systems. 19 Then the "TEWKDLR", "SITTDRL" and "DUNSDLR" and 20 "WYCODLR" refer to domains associated with De La Rue, so 21 there would be FTMS servers located within those four 22 sub-domains that were used to transfer data between 23 Pathway systems and De La Rue systems. 24 Q. So, for present purposes, really, we're only concerned 25 with the Pathway one at the top. Where it says "NT, All 130 1 Servers", when you reviewed this document what did you 2 think that covered in terms of scope? 3 A. That would be all NT platforms that were members of any 4 Windows NT domain. 5 Q. So it would include post office counters? 6 A. No, counters were excluded. These are Windows NT 7 platforms that are located in the data centres, Wigan 8 and Bootle. 9 Q. So Mr D'Alvarez in evidence yesterday said that it would 10 include counters. You disagree with that? 11 A. Yes, I disagree with that. 12 Q. In the -- well, we can go there, I think. Bear with me, 13 sorry. 14 Actually, I think it will be hard to bring the 15 document up, but in the security functional 16 specification, which was shown yesterday, one of the 17 points that was made was that in Riposte, a Windows NT 18 workstation was described as a message server, Riposte 19 message server. Is your evidence that that -- because 20 of that, this table -- it doesn't mean a counter falls 21 within this table "All servers"? 22 A. That's right. The counters were excluded from the 23 Windows NT domain design. 24 Q. But this does include the correspondence server? 25 A. The correspondence servers and the agent servers would 131 1 have been catered for within this design. 2 Q. Do you have any recollection as to what the Tivoli 3 remote console did? 4 A. My recollection of the Tivoli remote console was that it 5 was -- that, if you like -- the device that was used to 6 apply changes to the data centre systems. 7 Q. To the data centre systems, was that, sorry? 8 A. Yes. 9 Q. It's right, isn't it, that it would also be used to make 10 changes to counters? 11 A. I have no recollection of it being used for counters, 12 but my focus and my work was concentrated on delivering 13 for the data centres. 14 Q. So it wasn't within your remit to look at how the SSC 15 may or may not be able to access counter computers? 16 A. Well, I translated the document that Alan has authored 17 here, which is shown on the screen, so the definition of 18 the group name "SSC Apps MAN", I took that information 19 and I created Windows NT scripts that would enable that 20 tool set that's listed in the second column on the left 21 under "Tools", so that those tools could be made 22 available to the secure role, SSC Apps MAN, and I would 23 have also created the access control lists, which would 24 have enabled rewrite and execute to the folders on the 25 various platforms described under "NT servers". 132 1 Q. Do you know who was responsible for access control as 2 between the SSC, or anyone in Pathway for that matter, 3 and the counter branch computers themselves? 4 A. No. No, I'm not aware of the security fit that was 5 applied to counters. 6 Q. Given you were based in the security team and you were 7 involved in access control in some way, does that -- can 8 we infer that no one was overseeing that aspect, namely 9 who had access between the SSC and the counter systems? 10 A. I have no knowledge of anybody that was applying the 11 same controls that I was applying to the Windows NT 12 servers that were located in the data centres. 13 Q. Please could I go to FUJ00088036. This is a document 14 that's been shown a few times this week. It's the 15 "Secure Support System Outline Design" dated 16 2 August 2002. Do you recall if you saw this document 17 in 2002? 18 A. I'm not listed as a reviewer of that document, but I do 19 recall Geoffrey Vane. I do recall working with Geoffrey 20 Vane. 21 Q. Sorry, Geoffrey? 22 A. Geoffrey Vane. 23 Q. Why is that relevant to this document, sorry? 24 A. I believe that Geoffrey Vane, as the security TDA at the 25 time, would have been involved in this document. 133 1 Q. If we could turn the page just to see the distribution 2 list. So yes, Geoffrey Vane, security TDA. Do you 3 recall being -- well, let's first go to page 9, if 4 I may, so we can set out what the document did. So SFS, 5 do you know what that refers to? 6 A. I believe that refers to the security functional 7 specification. 8 Q. So the security functional specification: 9 "... mandates the use of Tivoli Remote Console ... 10 for the remote administration of Data Centre platforms. 11 This records an auditable trail of log-ins to all boxes 12 accessed by the user. It is a matter of considerable 13 discussion and correspondence that TRC is slow and 14 difficult to administer. This has led over time to BOC 15 personnel relying heavily on the use of unauthorised 16 tools (predominantly Rclient) to remotely administer the 17 live estate." 18 With that context, do you recall having discussions 19 about this document, or the issues raised in it in 2002? 20 A. I don't recall the discussions in terms of the detail 21 that's included within this document, but I do remember 22 the PWYSAS domain being created and the -- 23 Q. Sorry, could you just explain what that is, sorry? 24 A. So that would be Pathway secure access server, or 25 servers, and then within that sub-domain I remember the 134 1 MBOSAS01(?) and MWESAS01(?) so those would be been the 2 servers that were hosting the terminal services. 3 Q. What was the purpose -- 4 A. (Unclear) designed the first two. 5 Q. So those domains were created to essentially give effect 6 to the tool proposed here? 7 A. Yes. 8 Q. And you were involved in the creation of those? 9 A. I believe so, yes. 10 Q. Could we please turn to page 19. If you scroll down 11 a little, this diagram -- is that explaining the tool 12 that is being proposed to allow audited access to the 13 counter systems? 14 A. Yes. I think that diagram there is describing what we 15 knew as the Cygwin solution. 16 Q. And could you just explain what that solution entailed? 17 A. There was a terminal server, as described on the 18 right-hand side. On the secure access server there was 19 an SSH client and a similar SSH component logged on -- 20 Q. What's the SSH client? 21 A. It was a third party solution that enabled secure 22 connection between a client and a server and it enabled 23 certain functions, such as the capture of user 24 authentication and the auditing of that log on. 25 Q. So was this to be used by members of the support 135 1 services essentially to remotely access counters? 2 A. I believe so, yes. 3 Q. And you say it had -- well, we can see from the document 4 that there were enhanced, or more audit features 5 available, is that correct? 6 A. Yes. 7 Q. Do you know what data was captured for audit purposes? 8 A. No. 9 Q. Was audit not within your domain? 10 A. I didn't need to know the detail of data records that 11 were being captured. What I needed to do in that role 12 was to make sure that the development team, looking 13 after and making changes to the audit solution, were 14 aware of those changes that they needed to code and my 15 role was to report to people like Pete Dreweatt and 16 Ian Morrison whether the audit team was on track or not. 17 Q. So if we can go further down on this document please 18 there should be a small -- I think it's over the next 19 page, sorry. Thank you. This is referring to the 20 secure support access server, SSAS. 21 A. Yes. 22 Q. Can you just assist with what that is? 23 A. Well, that is the secure access server that I was 24 referring to earlier. 25 Q. That's what you were referring to? 136 1 A. Yes, yes. 2 Q. And the additional security domain, it says: 3 "Third line and Operational Support Units have, and 4 require, system admin access to all systems they manage. 5 In order to create a non-refutable audit of all actions 6 carried out against systems they manage it is necessary 7 to restrict their access to the system which is 8 gathering the audit." 9 So in essence is that saying support services 10 shouldn't be able to access the audit data captured 11 through this tool? 12 A. Yes. 13 Q. It goes on -- we don't need to read it all -- to refer 14 to a second security domain being required and changes 15 will be required to the existing security NT domain 16 hierarchy. In the box it says: 17 "Question for Mark Ascott: Could this be as simple 18 as moving the administration of the PWYDCS user domain 19 to a separate group?" 20 Could you please just elaborate on what that 21 question meant? 22 A. I think that question is exploring, you know, is there 23 a simple change to the NT domain structure, or, you 24 know, is there a more involved change. The person 25 asking that question, whoever wrote that design note, is 137 1 clearly looking to me to provide some guidance as to how 2 simple or quickly the change can be made, or, you know, 3 is there more complexity required. 4 Q. And do you recall when this change was made? 5 A. I think this is part of the conversations and the work 6 that I would have done with Geoffrey Vane in order to 7 bring this in, this change in. As I say, for me my 8 recollection is that I recommended that we introduced 9 the PWYSAS domain and placed the secure access servers 10 as member servers of that specific domain, that 11 sub-domain. 12 Q. When this tool was developed, presumably it required 13 testing? 14 A. Yes, it would have done. 15 Q. Were you involved in that side of things? 16 A. I would have been involved in delivering the changes 17 that introduced the PWYSAS domain and setting up the 18 secure role users and the secure profiles, the access 19 controllers. My scripts would have been passed to the 20 PIT team, as it was known then, the product integration 21 team, and the first real test for those scripts would 22 have been when the SPTS team, the service provision and 23 test services group -- they were a small team, they were 24 located in Feltham, and they would have used those 25 scripts after the PIT team had wrapped around 138 1 a deployable script in to enable the software deployment 2 tools to execute those scripts. So they would have been 3 involved in the testing for real, in terms of did those 4 scripts enable the PWYSAS domain to be added to the data 5 centre solution. 6 Q. But do you have any recollection of the results of the 7 testing related to this tool? 8 A. No, no. So in my own testing of my own scripts I would 9 have desk checked them. I wouldn't have had available 10 to me an equivalent test rig of all of the NT systems 11 and all of the NT domains, so I would have desk checked 12 and I think there was a technique that I used to use 13 where I could execute the scripts but using Echo to 14 print, if you like, the commands that were being 15 executed and I would have received those responses back 16 and I would have then been able to tell whether the 17 logic in the script was doing what I intended it to do. 18 Q. But that was all prior to going for formal testing? 19 A. That's right, so that was my own development testing 20 that I was doing to satisfy myself that the changes were 21 ready to be made available to the product integration 22 team. 23 Q. And as part of your role in developing this tool and as 24 part of your role in audit, did you have any oversight 25 as to whether or not this tool was being used for the 139 1 purpose of remotely accessing counters? 2 A. So I'm providing the basis for the platform to be part 3 of the overall domain structure. The application -- the 4 SSH application that's sitting on top of that is a tool 5 that I'm enabling the secure role and the users that are 6 created from that secure role to execute. The 7 configuration of the SSH tool itself, I was not 8 delivering that configuration. 9 Q. I want to ask a few points about something that was 10 raised in evidence with Graham Allen yesterday. Did you 11 listen to his evidence? 12 A. I did see some of Graham's recording, yes. 13 Q. He said that -- I'm paraphrasing, but that the team 14 developing Horizon Online was not able to look at the 15 code for Legacy Horizon for intellectual property 16 reasons, is that right? 17 A. Yes. I remember in 2008 when I rejoined Pathway in the 18 test team that there was a lot of emphasis put on the 19 counter team not reviewing or going anywhere near what 20 was the Riposte solution for fear of claims that, 21 you know, the new HNG-X counter that was being developed 22 was a copy, or was, you know, a facsimile of the Riposte 23 system and that was a message that was regularly 24 mentioned, certainly in 2008. You know, although the 25 counter development team was located in the Braro 1 140 1 Tower(?) and I was primarily working in the test team 2 lab area on the ground floor, the ripples of those types 3 of reinforcing that message would reach down into the 4 test teams. 5 Q. Are you aware of whether that inability to look at the 6 Legacy Horizon code caused any difficulty, or any -- it 7 held things up when developing Horizon Online? 8 A. Again, I wasn't part of the counter development team, so 9 I can't really say how they went about solving the 10 problems of delivering and developing the new HNG-X 11 counter. You know, we were very much aware that, 12 you know, there was this -- it didn't feel like 13 a directive, you know, "You will not go and look at that 14 previous counter design or that previous counter code", 15 but it wasn't far away from that. The direction was 16 relatively clear that, you know, we could -- well, 17 ICL/Fujitsu could be taken to task for copying Riposte, 18 so ... 19 Q. In your role and when working with Legacy Horizon did 20 you ever have to deal with third party software 21 suppliers? 22 A. Can you repeat the question again please? 23 Q. Yes, of course. Let me put it another way. I know you 24 didn't work with Riposte, but that was an application 25 provided by Escher -- 141 1 A. It was. 2 Q. -- a third party, and the question I ask is: in your 3 roles, did you have something similar where you had to 4 work with a third party outside of Pathway in respect of 5 an application you oversaw within Legacy Horizon? 6 A. Well, I would say no to that. I mean as part of 7 developing the secure role templates, those templates 8 needed to refer to applications and executable files 9 from a range of applications, but, you know, what I was 10 processing was, you know, "This is the tool, this is the 11 executable -- this is where you can expect that to be 12 located on the NT server or the NT work station", so if 13 I knew the location of the executable I could configure 14 that into the secure role template so -- 15 Q. So no equivalent then to working with Escher? 16 A. I didn't have to have any dealings with any third party 17 suppliers. In Horizon Online obviously, as part of the 18 volume and integrity test activity, we utilised the 19 Hewlett Packard LoadRunner tool set and in order to use 20 that, so tools and applications, we needed to have 21 access to their support desk, you know, when we 22 encountered issues with using that tool, but that was 23 solely for Horizon Online and the HNG-X solution. 24 MR STEVENS: Sir, I wonder if that might be an appropriate 25 time to have a break? 142 1 SIR WYN WILLIAMS: Yes, it would. Sorry about the slight 2 delay but I was on mute. 3 MR STEVENS: Thank you, sir. 4 SIR WYN WILLIAMS: So what time shall we start? 5 MR STEVENS: Quarter past, would that work? 6 SIR WYN WILLIAMS: That's fine. 7 MR STEVENS: Thank you. 8 (3.03 pm) 9 (Short Break) 10 (3.14 pm) 11 MR STEVENS: Good afternoon, sir, can you see and hear me? 12 SIR WYN WILLIAMS: Yes, I can. Thank you. 13 MR STEVENS: Mr Ascott, I only have one question really. 14 You said at the beginning of your evidence, towards the 15 start, that when I referred to the 2009 Computer Weekly 16 article, you thought that publications or journalists 17 were biased against ICL, yes? 18 A. Yes. 19 Q. Do you still hold that view, or have you changed your 20 mind knowing what you know now about Legacy Horizon? 21 A. What I know now from news media organisations like the 22 BBC, ITV, Sky, I would say yes obviously I am aware that 23 prosecutions were made that not necessarily should have 24 been, but at that time I had had an experience whereby 25 a journalist wrote for our local newspaper that 143 1 a teenage girl who had been glue sniffing -- 2 Q. Sorry, is this relevant to -- 3 A. Well, it sets my mindset in relation to articles 4 produced and written by journalists, so for me, 5 you know, my experience in that instance ... I take with 6 a pinch of salt what journalists write. 7 Q. But, as I say, knowing what you know now, what is your 8 view of the Computer Weekly article? 9 A. Yes, knowing from the court cases and the judgments that 10 have been passed, now I know differently. 11 MR STEVENS: Sir, I have no further questions, but 12 I understand that there are -- I will just check if 13 there are any questions in the room, sir. 14 (Pause) 15 No, it's a nil return, sir. 16 SIR WYN WILLIAMS: There we are, Mr Ascott. The questions 17 that have already been put to you are the ones that 18 people want to put, so thank you very much for coming to 19 give your evidence and for answering the questions and 20 thank you for making your written statement as well. 21 So I take it that concludes this afternoon's 22 business, Mr Stevens? 23 MR STEVENS: That does, sir. We will be back at 10 o'clock 24 tomorrow if that suits you. 25 SIR WYN WILLIAMS: Yes, of course. So 10 o'clock tomorrow 144 1 morning. 2 MR STEVENS: Thank you, sir. 3 (3.17 pm) 4 (The Inquiry adjourned until 10.00 am on Thursday, 5 10 November 2022) 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 145 1 2 INDEX 3 4 JOHN SIMPKINS (affirmed) .............................1 5 Questioned by MR BEER ............................1 6 Questioned by MS PAGE ...........................86 7 Questioned by MS PATRICK ........................97 8 MARK ASCOTT (sworn) ................................113 9 Questioned by MR STEVENS .......................113 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 146