1 Thursday, 10 November 2022 2 (9.59 am) 3 MR BLAKE: Good morning, sir. 4 SIR WYN WILLIAMS: Good morning. 5 MR BLAKE: Today's first witness is Mr Jarosz. 6 MARK JAROSZ (sworn) 7 Questioned by MR BLAKE 8 MR BLAKE: Good morning, could you give your full name, 9 please? 10 A. Mark Jarosz. 11 Q. Mr Jarosz, you should have in front of you a bundle 12 containing a witness statement. Is that statement dated 13 9 August of this year? 14 A. Yes, I have. 15 Q. Yes. Could I ask you to turn to the final page, page 21 16 of 22, and is that your signature there? 17 A. Yes, it is. 18 Q. Thank you. Is that statement true to the best of your 19 knowledge and belief? 20 A. Yes, it is. 21 Q. Thank you, Mr Jarosz. I'm going to ask questions on 22 behalf of the Inquiry today. Thank you very much for 23 attending today and thank you for your witness 24 statement. The witness statement, for the purposes of 25 the transcript, is WITN04810100. That statement and the 1 1 exhibits will all go into evidence and what I'm going to 2 ask you today will build upon what's already in there. 3 I think we will probably be half a day, possibly less. 4 I'm going to start with your background. You joined 5 ICL in 1983; is that right? 6 A. Yes, it is. 7 Q. You became employed by ICL Pathway or what became 8 ICL Pathway? 9 A. Yes. 10 Q. You started as a customer support executive? 11 A. I did, yes. 12 Q. Then you became involved in Horizon from 1995 to 2012; 13 is that right? 14 A. Yes. 15 Q. So you were involved before even ICL had succeeded in 16 the procurement exercise? 17 A. Yes, I was. 18 Q. In 1996, you became a solution architect networking; is 19 that right? 20 A. Yes, it is. That was my role, yes. 21 Q. That was your role, and that was Legacy Horizon, or 22 what's now known as Legacy Horizon? 23 A. Yes. 24 Q. Then in 2010 to 2012 you were a solution architect 25 security, was that in relation to Horizon Online? 2 1 A. Yes, it was. 2 Q. You are still employed by Fujitsu; is that right? 3 A. I am, yes. 4 Q. Your title now is lead domain architect? 5 A. That's correct. 6 Q. Presumably, you still have access, therefore, to Fujitsu 7 records and things like that? 8 A. Yes, partially I do, yes. 9 Q. You are represented today and assisted by the Fujitsu 10 legal team? 11 A. Yes, I am. 12 Q. Can we bring up on screen your witness statement. It's 13 WITN04810100, please. Now, this statement was provided 14 in response to a Rule 9 request to Fujitsu on 11 March 15 of this year for a corporate statement relating to 16 phase 2 of the Inquiry. Are you aware of that? 17 A. I am, yes. 18 Q. You were chosen by Fujitsu to be one of several 19 witnesses who we have heard from to respond to that 20 request. 21 A. Yes. 22 Q. Now, in that original Rule 9 request, there was 23 a section about robustness; do you remember that? 24 A. I do, yes. 25 Q. Yes, and it asked for an explanation as to what was 3 1 known by ICL about the accuracy and integrity of the 2 data recorded and processed on the Horizon System. 3 A. Yes. 4 Q. It also asked about the extent to which deficiencies in 5 the Horizon IT system were capable of causing or caused 6 apparent discrepancies or shortfalls; do you remember 7 that? 8 A. I do. 9 Q. After you gave a draft statement, you were sent a second 10 request asking for more detail in certain respects; do 11 you remember that? 12 A. I do. 13 Q. That request was 1 July of this year. Again, in that 14 request there was a broad question about robustness. 15 Do you remember that? 16 A. I do. 17 Q. You may not remember but, in that request, at the top, 18 there was a section in capital letters saying that you 19 are expected to have refreshed your memory from 20 contemporaneous documents. Do you remember that 21 section? 22 A. I do. 23 Q. Let's look at your statement. It begins with 24 an introduction. Could we scroll on to the next page 25 please. It then has a background section and, over the 4 1 page, it goes on to talk about the bid for the Horizon 2 project and, over the page, this is paragraph 12, and 3 there you say that there were a number of decisions 4 before you came onto the scene, one of which was the 5 ISDN decision, to use ISDN, and the second was to use 6 Riposte. Is that a fair summary of that paragraph? 7 A. Yes, it is. 8 Q. They are two big decisions that are mentioned in that 9 paragraph. You have said a number of decisions but, 10 presumably, you see those as the two significant 11 decisions that were taken before your time? 12 A. Yes, those were the two main ones. There are a few 13 further ones as well. 14 Q. The point that you make is that the decision to choose 15 Riposte was not your decision. 16 A. That's correct. 17 Q. Your initial role, I think, was to do performance 18 modelling on Riposte; is that right? 19 A. Yes, on Riposte and the network, yes. 20 Q. Can we look at paragraph 18, so we can scroll on 21 a little bit more. Thank you very much. At 22 paragraph 18, you say: 23 "At this initial stage, I did have some concerns 24 about whether the Riposte messaging solution would 25 effectively scale to approximately 20,000 branches, as 5 1 it had not been proven to work at that scale before. 2 This was not a concern that was unique to me, but was 3 a known issue that was actively discussed within the bid 4 team and with Escher." 5 Looking at paragraph 19, you say -- I will just turn 6 to my own copy: 7 "Managing the issue of scaling Riposte was not 8 within my ... responsibility. However, I do recall, 9 from my general involvement on the architecture team, 10 that this concern was eventually addressed in the 11 deployment phase (during and prior to the pilots and 12 rollout of Horizon)." 13 So again, what you are making clear there is that 14 that wasn't your responsibility, the scaling of Riposte, 15 but it was addressed? 16 A. Yes. I was very much aware of that. 17 Q. Can we look at paragraph 21, please. In that paragraph, 18 you set out the approach that had been taken to Riposte 19 and how it had been decided that it would operate. So, 20 again, it's emphasising there that that wasn't your 21 decision as to how to operate Riposte; is that correct? 22 A. That's correct. 23 Q. Paragraph 22, please. You say there you didn't have any 24 concerns about the use of Riposte in that manner. So, 25 again, it wasn't your decision how to use it but you 6 1 didn't have any concerns about its use in the manner in 2 which it was used; is that correct? 3 A. That's correct. 4 Q. Can we look at paragraph 24 and 25, please. 24: 5 "In order for this design to function on the Horizon 6 System, Escher needed to develop new software for use on 7 Riposte." 8 So 24 and 25, I think, explain the new software that 9 needed to be developed and then, scrolling over to 26, 10 it says there that you worked on the ISDN network 11 solution, so that was the focus of your work there; is 12 that right? 13 A. Yes, that's correct. 14 Q. Can we look over the page to paragraph 27, and you say, 15 in respect of the ISDN work that you carried out: 16 "... the bid team internally convinced ourselves 17 that the ISDN solution was sufficient." 18 A. Yes. 19 Q. So -- 20 A. Sorry. 21 Q. Sorry. 22 A. Yes, that's correct. It took a while to come to that 23 conclusion. 24 Q. Yes, so that's the area that you say you were 25 responsible for, the ISDN connection, and you were 7 1 ultimately convinced that it was sufficient; is that 2 right? 3 A. Yes. 4 Q. Paragraph 29, please, and onwards address the Initial Go 5 Live pilot. I think you highlight in that paragraph, or 6 in paragraph 31, that the Initial Go Live was limited 7 from your perspective because it had a permanent ISDN 8 connection, so it didn't test the more intermittent ISDN 9 connection. 10 A. Yes. 11 Q. But 32, so scrolling down, you didn't recollect any 12 specific problems that arose during that Initial Go Live 13 phase; is that right? 14 A. Yes, not within my area, which was the network area. 15 Q. Yes. Over the page, to the 200 to 300 branch pilots. 16 Again, you say there, in paragraph 34, you don't recall 17 any problems occurring; is that right? 18 A. Yes, that's correct, in my area, which was the network 19 at that stage. 20 Q. Then 35 onwards addresses the pilot and the rollout of 21 New Release 2. At paragraph 38, please, you observe: 22 "During the pilot, we observed a number of issues as 23 we worked towards scaling the Horizon solution", and you 24 set out there three issues. 25 I think (a) could be summarised as moving some 8 1 external storage; is that right? 2 A. Yes, it is. 3 Q. (b) is providing a VSAT to remote branches, so instead 4 of the ISDN certain branches could use a satellite 5 connection? 6 A. Yes, so that was dealing with the fact that ISDN, 7 although it was the primary network technology, wasn't 8 available everywhere, so there needed to be 9 an alternative solution. 10 Q. And (c), if we could keep on scrolling to (c), software 11 updates needed to be scheduled differently because they 12 were all taking place at the same time and causing some 13 difficulties; is that right? 14 A. Yes. 15 Q. Then we go to paragraph 40, please, where you say: 16 "Beyond the points above, I do not recall the issues 17 that arose during the NR2 pilot. However, I believe 18 they were ... typical of [any] large-scale IT projects 19 of the time." 20 You don't recall any particular issues that 21 contributed to the delay of the NR2 pilot or the rollout 22 of the system. 23 It is paragraph 46 then that addresses the issue of 24 robustness and I'm going to read that paragraph. It 25 says: 9 1 "I am aware of the Inquiry's definition of 2 'robustness'. I am only able to evaluate the Horizon 3 system's robustness from the perspective of my roles on 4 networking and security, and I note that I had a much 5 more limited involvement in relation to Horizon Online 6 than its predecessor." 7 Just to be clear, there is a section in your 8 statement on Horizon Online that I have skipped over. 9 A. Yes. 10 Q. "It was also not my role to design or develop the 11 applications that would have recorded/processed data on 12 Horizon, including in relation to branch accounts. From 13 that perspective, I did not have concerns about the 14 robustness of Horizon, nor was I aware of any." 15 Can I just clarify, was there another Mark Jarosz 16 working at ICL in 2000/2001. It's a pretty unique name, 17 presumably you were the only Mark Jarosz? 18 A. Only one, yes. 19 Q. You have been given some papers over the past few days, 20 many of which with your name on, which relate to Riposte 21 bugs, what's known as "Riposte lock" -- commonly 22 referred to as "Riposte lock", and that is known to have 23 fed into what we know as the Callendar Square bug. 24 Which paragraph of your statement do we find mention of 25 the Riposte lock issues? 10 1 A. So in terms of the Riposte lock issues, the reason I was 2 involved in that was because the people working on the 3 problem needed to find out from Escher what the error 4 messages meant and, at the time, there were a very few 5 of us who had a working relationship with Escher. So my 6 role was to ask questions directly, face-to-face with 7 Andrew Sutherland from Escher about what that meant and 8 convey his response to the people working on the problem 9 in ICL at the time. 10 Q. Yes, and where in your statement can we find reference 11 to the Riposte lock problem with Horizon? 12 A. I didn't mention the Riposte lock problem in my 13 statement. 14 Q. Did you follow the Group Litigation, the Bates and 15 Others case, did you follow that at all? 16 A. In the press as it was reported, yes. 17 Q. So you still work for Fujitsu, so presumably it's quite 18 well-known? 19 A. Yes. 20 Q. Did you, presumably, understand the significance of 21 those Riposte lock events in the context of that case? 22 A. No, sorry, I didn't. 23 Q. Did you follow the Callendar Square incident at all? 24 A. No, sorry, I didn't. 25 Q. I'm going to take you to the documents in a moment but 11 1 it looks from those documents that you were quite 2 a central figure in trying to resolve or deal with 3 Escher in relation to that Riposte lock problem. Is 4 that a fair description of your role? 5 A. Well, I was working with Escher at the time on the 6 networking aspects of Riposte, which meant I spent time 7 in their facilities in Boston, USA, and when people 8 working on such issues had questions of them then, 9 because there wasn't much documentation, to the best of 10 my knowledge, about the Riposte -- the messaging 11 product, the way the questions were resolved was to ask 12 them directly, face-to-face and, whilst it was the case 13 that, during the bid phase, Escher did attend ICL 14 offices in Feltham, at that stage, they were mainly in 15 Cambridge, Massachusetts. So my role was to convey 16 those questions directly to Escher and get responses and 17 feed those back. 18 Q. So you were being given problems by engineers working on 19 particular problems and your role was the direct liaison 20 with Escher in relation to those problems? 21 A. Yes. There were other people, not just me, involved in 22 the liaison but not many and I was one of them. 23 Q. Yes. I mean, it's fair to say from that that you were 24 fairly involved in trying to resolve Escher-related bugs 25 in that case, weren't you? 12 1 A. Well, as one of the examples shows, my role was to 2 convey the information back to our teams so they could 3 progress with what they were doing. In many cases, the 4 information I provided was not sufficient for them to 5 resolve the bug but allow them to progress with it. 6 Q. So is your evidence that you were simply the liaison 7 with Escher -- 8 A. In that particular example of -- 9 Q. -- and you weren't making decisions -- I mean, similar 10 to the other parts of your evidence, where you say 11 "Decisions were taken and I was simply following them"; 12 is that the position in relation to Riposte lock? 13 A. In the example that you gave, Riposte lock, that was the 14 case. There are other examples which were also in the 15 pack, where I was asked by the architecture group to 16 take a more proactive role. 17 Q. But in Riposte lock you didn't take a proactive role? 18 A. No. 19 Q. And there are other bugs that you did take a proactive 20 role in relation to? 21 A. Yes. 22 Q. Where are those mentioned in your witness statement? 23 A. So the example was a Riposte bug and I didn't mention it 24 in my witness statement. This is -- I think it is E1, 25 it was called the "handle leak problem". 13 1 Q. We will look at the handle leak problem as a background. 2 Can I just ask while we are on this issue -- we can take 3 down the witness statement, thank you -- what was your 4 relationship with Gareth Jenkins at this particular 5 time? 6 A. So I would describe it as professional, based on the 7 need to work together, because we were part of the -- at 8 the time, Alan Ward's team, so Gareth would -- when 9 Gareth was aware, for example, that I was going to visit 10 Escher, he may ask me some questions to convey to them. 11 Q. Were you senior to him; at the same level? 12 A. Same level. We worked in -- we had different 13 responsibilities within the architecture team, but we 14 were level. 15 Q. We will go to the correspondence in due course, but it 16 looks, from some of that correspondence, that he is 17 looking to you for guidance; would you accept that? 18 A. No, because he was a peer working at a different part of 19 the solution. So whilst I was responsible for the 20 networking part of the solution, he was responsible for 21 the counter and agent applications. 22 Q. Would you say you had joint responsibility then for 23 certain issues? 24 A. Well, I can imagine that could arise, yes, where there 25 was an issue where it wasn't clear where the issue lay. 14 1 Q. I mean, something like the Riposte lock problem, would 2 you have joint responsibility for that? 3 A. Well, no, because, in that particular example, what 4 Gareth wanted to know from Escher was what that error 5 message meant. The Riposte product logged lots of error 6 messages and there was no documentation which said what 7 this error message means and what the consequences could 8 be, so he needed someone to ask that question and, in 9 some cases, he asked me; in other cases he would have 10 asked the liaison that was at Escher, because we had 11 people who were there on secondment to act in that 12 liaison role. 13 Q. So, again, you were the conduit rather than the person 14 who was responsible? 15 A. Yes, one of them, yes. 16 Q. Were you ever asked to give statements in criminal 17 proceedings? 18 A. No. 19 Q. Were you ever involved in who would give such 20 a statement? 21 A. No. 22 Q. As peers, why was Gareth Jenkins selected and you 23 weren't; do you know? 24 A. I don't have knowledge of why that was. 25 Q. Were you ever involved in researching historic issues 15 1 with Riposte, more recently, for example? 2 A. No. 3 Q. I'm going to take you to a document, it's POL00028911. 4 This is a document that we may well come back to and 5 I don't think it's necessarily a document you have seen. 6 Is it a document that you are familiar with at all? 7 A. No, I don't recognise that document. 8 Q. So the only relevance, for current purposes, are that it 9 concerns the Callendar Square bug and, if you look at 10 the list of PEAKs, it lists the PEAKs that are related 11 to that issue, and one of them is PC0056922, and that's 12 something that we're going to come back to in due 13 course. So we can take that document down for now, but 14 we will look at that particular PEAK. 15 Let's look at the contemporaneous documents from 16 2000/2001. Can we look at FUJ00078274, please. So this 17 is going to be a bit of background before we get to the 18 particular PEAK. This is an ICL "Weekly Progress 19 Report" for 30 July 2000 to 2 August 2000. Can we look 20 at page 3, please. 21 So this is a document you are familiar with and 22 I think you have already referred to one of the issues 23 that's raised there and let's have a look at those. Can 24 we scroll down that page, please -- a little bit more, 25 so that we have the whole of that 1.2 in view, please? 16 1 So here there are two major critical issues arising 2 during the week. The first, handles leaks in the 3 Riposte message server which could ultimately threaten 4 rollout if not resolved and it says "An urgent fix is 5 being sought from Escher". That's the one you referred 6 to just a moment ago, is it? 7 A. Yes, it is. 8 Q. Again, that one isn't mentioned in your statement, is 9 it? 10 A. No, it isn't, but -- 11 Q. Can you very briefly explain what that relates to, the 12 leaks in the Riposte message server? 13 A. Yes, so during -- I believe during testing, it was 14 observed that some resources used by the Riposte message 15 server were increasing and the testers were concerned 16 that that behaviour suggested there was a leak in the 17 Riposte message server. 18 Q. What does a leak -- what does that mean? 19 A. It means that it's using resources in a manner that 20 eventually it will run out of resources and stop 21 working. So that was the interim conclusion reached by 22 tests and, therefore, it raised quite a few concerns. 23 So my role was to ask -- initially -- this was agreed 24 within the architecture team -- was to describe the 25 scenario to Escher and ask them whether this was a bug 17 1 or behaviour as designed. 2 They confirmed it was -- Andrew Sutherland confirmed 3 this was behaviour as designed, so within the 4 architecture group we then decided to see -- and, by the 5 way, Andrew Sutherland also explained to me why this was 6 happening and when it would stop. 7 Q. Can I just ask, who is Andrew Sutherland? 8 A. He is the chief architect for the Escher group messaging 9 product. So he is the kind of person who knows about 10 the product the most. 11 Q. Would he be your direct liaison with the Escher group? 12 A. Yes. 13 Q. There's a second problem that's mentioned there. The 14 second problem is the failure to swap out slave counters 15 on -- we have seen this before, is it "CI4"? Is that 16 something you remember, or is it "Cl4", "C14"? 17 A. I think it is "CI4" but I just -- I remember it as being 18 one of the releases that we were doing. 19 Q. Yes, and it says: 20 "At present, intermittent fault causes the Riposte 21 service to hang." 22 It continues: 23 "Investigations of slave swaps has shown the problem 24 occurring at a number of different points in the process 25 of copying the squirelled message store", et cetera. 18 1 Can you briefly explain what that issue was at all? 2 A. No, I wasn't involved in that, so I -- I wasn't asked to 3 help with that issue. 4 Q. Is this a document that you would have seen at the time 5 though, ICL weekly progress report? 6 A. Well, I may have received it on an email but I can't 7 remember reading it. 8 Q. I mean, do you remember receiving Pathway weekly 9 progress reports in 2000? 10 A. I do recall being copied on them, yes. 11 Q. Would it not have been of interest to you? 12 A. So, yes, I would be interested, if there were network 13 issues, and in the issue that -- the handle leak issue, 14 it was called to my attention, so I was involved in 15 dealing with it. 16 Q. Are you able to assist us with what it means by 17 "intermittent fault causes the Riposte service to hang"? 18 Is that a lock issue or is that something else? 19 A. I can speculate what that means, in general terms, 20 because, if Riposte is hanging, I would assume it means 21 it is unresponsive and can't be used for anything and 22 needs to be restarted. 23 Q. Can we look at page 6, please. At the bottom, there is 24 a section on "Current Critical Problems", and there are 25 the two problems there that we have just discussed. The 19 1 first is getting the squirelled message store, they 2 can't successfully swap out a faulty counter on CI4, and 3 then the second one is the issue "in live with handle 4 leak", and it says there: 5 "Gareth Jenkins will address this issue. In the 6 meanwhile Mark Jarosz will liaise with Escher to 7 establish the root cause of the leak." 8 A. Yes, so just to confirm, that's exactly what I did: 9 I liaised with Escher and I fed back my findings to the 10 team internally within ICL. As a result of that, 11 because Riposte was working as designed, based on the 12 feedback, the decision was made to attempt to reproduce 13 the problem, or reproduce the scenario, in our test 14 facilities in Bracknell where we had the ability to 15 simulate thousands of counters connecting to 16 correspondence servers, and that proved that this was 17 not an issue. 18 Q. Now, as I said, this is -- I'm taking you to this for 19 background and to establish the roles and 20 responsibilities. 21 A. Yes. 22 Q. It seems as though Gareth Jenkins and yourself are the 23 prime, principal contacts with regards to Riposte 24 errors, at that stage; is that right? 25 A. So Gareth Jenkins' role in this was based on the 20 1 assumption that this is an issue that needs to be 2 addressed and how we would mitigate that in the live 3 solution. My role with the performance team was to find 4 out if we needed a fix from Escher or whether this was 5 working as designed. 6 Q. I mean, what you're doing: you're not just kind of 7 passing messages to Escher though, are you? You're 8 described here as establishing the root cause of the 9 leak, or working with Escher to establish the root 10 cause. 11 A. Yes, but, in this particular example, it -- a very brief 12 conversation with Andrew Sutherland confirmed that there 13 was no problem, so the assertion there was a leak was 14 incorrect and, in order to test that, we -- that's why 15 we ran it on this test facility we had in Bracknell to 16 confirm all was -- there was no problem. 17 Q. I'm not concerned with the particular issue that 18 occurred here. I'm more concerned about the different 19 roles and responsibilities. 20 A. Okay. 21 Q. Certainly reading here, you are acting as more than just 22 simply a messenger with Escher; you are the person who 23 is liaising with them, in order to find out the root 24 cause of the problem? 25 A. That's very true, yes. 21 1 Q. Was that typical of your job? 2 A. I can only recall a few issues that I was asked to look 3 at, which are of this significance to the programme, and 4 this is one of them. So, no, it wasn't typical. My 5 normal day job was the evolution of the network, which 6 also included changes to Riposte to work over the 7 network. 8 Q. Would it be typical for Gareth Jenkins to be working on 9 the technical side of something and for him to ask you 10 to liaise with Escher to try and resolve it? 11 A. Well, there are examples where he has done that, yes, 12 but typically by email, but the -- what he asked me to 13 do was to ask specific information of Escher and, 14 typically, that would have been there's some 15 observations made based on error messages and what do 16 they mean, if that was not already known to him. 17 Q. Typically to establish the root cause of a problem? 18 A. Yes, partly problem investigation. 19 Q. Can we look at FUJ00083544, please. Thank you very 20 much. 21 Now, this is the PinICL that I mentioned earlier and 22 that was mentioned in that Callendar Square document. 23 The PinICL itself is at the bottom, it has been 24 forwarded, and it is PinICL 56922. Can you see that? 25 The title, in the subject at the bottom? 22 1 A. I can, yes. 2 Q. Thank you. Can we go over the page to page 2, please. 3 I'm going to take some time over this document. Can we 4 scroll down slightly on this page. There is an entry at 5 19.15 on 1 November. Yes, it's the fourth entry there, 6 and it says: 7 "PM [that's postmaster] reports error message when 8 trying to redeclare her cash." 9 Thank you. It says -- there's another entry there: 10 "Guided caller thru redeclaration: 11 "STK ..." 12 Do you understand what it's saying there, just that 13 entry "STK bal/dec cash ..." 14 "Dec" may be December, perhaps? I don't know, it 15 may not be. 16 A. I'm not 100 per cent sure what the abbreviations mean, 17 whether it's referring to the navigation on this 18 counter, I ... 19 Q. "Error message says 'error committing declarations' 20 "Voiced call to Dave in smc ..." 21 "SMC"? 22 A. I think that's one of our support teams. 23 Q. Yes: 24 "... who requested I pass the call over to them. 25 Caller [advised] and ref [number] given." 23 1 Then it says: 2 "User 'ADA001' advises that when a SU (CASH) 3 declaration is made the declaration would not be 4 accepted -- "searched kel for Error committing" -- 5 nothing. 6 "Searched events from web PAGE for counter 1 -- 'An 7 unexpected error occurred [while] attempting to modify 8 an entry in the run map. Timeout occurred waiting for 9 lock' and also critical 'Error Number ...'" 10 It gives an error number, et cetera: 11 "The Riposte PutObject function call returned 12 an error -- this happened while", et cetera, et cetera. 13 Then we go down the page and it shows that at 22.16, 14 so that's near the bottom of the page: 15 "Repeat Call: [postmaster] is still waiting for 16 a phone call it has been three hours since this issue 17 arose. Please ring immediately. 18 "The [postmaster] is only still available due to 19 living on the property." 20 Can we go over the page, please. The first 21 substantive entry there is 2 November still, 9.24: 22 "as pm [postmaster] is trying to redeclare cash to 23 alter she is getting error in declaration of cash 24 declaration error in committing list. 25 "Pm tried to create a new declaration for the 24 1 difference and got the same message." 2 Do you understand that at all? 3 A. Well, in general terms, I understand that these are 4 operations being performed on the counter, yes. 5 Q. Is this an example -- I don't know -- of a postmaster 6 trying to re-enter a declaration because of the problem 7 they are experiencing? 8 A. It is hard for me to say because I'm not familiar with 9 the counter application and how it's used. 10 Q. Okay. Let's move down, please, and it says there -- it 11 is the entry about halfway down the page, or 12 three-quarters of the way down: 13 "The above kel outlines the problem ... 14 "HSH1 Information: 15 "Called [postmaster] on the [advice] of Sara in smc 16 to get the messages [postmaster] is getting, 17 [postmaster] would like call back as is now trading 18 manually and is not being called back to get problem 19 solved." 20 So it looks as though the postmaster there has 21 stopped using Horizon and is trading manually. Do you 22 agree with that interpretation? 23 A. Yes. 24 Q. Then slightly below, 9.38, if we could scroll down 25 a little bit, it says: 25 1 "The call summary has ... changed from: 2 "PM reports error message when trying to redeclare." 3 It is now: 4 "... error committing declarations." 5 Is that something you understand at all? 6 A. No, I'm sorry, I don't. 7 Q. Could we go over the page, please. There's an entry at 8 9.40 on the next page, and it says there: 9 "This call has been raised to 'A' as [Post Office] 10 is manual due to being unable to roll over SU due to 11 events being generated by gateway which SSC are 12 actioning as per KEL." 13 It has effectively been given an "A" priority: 14 "Mike Woolgar rang in. I explained situation and he 15 requested that he be paged again if situation not 16 resolved by 13.00." 17 Can we go down to 10.30, please. It seems there: 18 "nbsc chasing ..." 19 It's a priority call: 20 "nbsc say [postmaster] is on manual, [postmaster] 21 was called this morning by 2nd line and told nonsense. 22 [Postmaster] is very angry and feels that she is being 23 messed about. Contacted edsc who states that haven't 24 called pm. Called smc is checking with the person who 25 was dealing whether they called [postmaster] will call 26 1 back. Nbsc says will call back in 20 minutes if no 2 resolution." 3 Were you, at that time, familiar with these kinds of 4 concerns from postmasters? 5 A. No. 6 Q. 10.36, the entry there says: 7 "If nbsc ring back on this call please contact an 8 stsa. Has given a 20 minute deadline in which she is 9 calling us back." 10 10.46, slightly further down the page: 11 "Spoke to Les -- passing call over urgently. 12 Advised user to reboot as she was stuck in a loop ... 13 and contact NBSC as to extending [Cash Accounting 14 Period]. Message store and Event log audit logs 15 coming." 16 Now, were you aware, or are you now aware that 17 a workaround in relation to this problem was rebooting? 18 A. Well, I'm now aware that's been mentioned, but the -- 19 Q. Do you remember your state of knowledge about the 20 Riposte lock issue and whether a workaround was, at that 21 time, to reboot? 22 A. No, and that wasn't the advice that was given, that 23 I recall from Andrew Sutherland either. 24 Q. But you would accept that that is the advice that's 25 being given in this particular PinICL, "Advised user to 27 1 reboot as she was stuck in a loop"? 2 A. Yes, I mean, it's very clear, yes. 3 Q. Can we go over the page, please, and it's about halfway 4 down the page, 11.22. It says: 5 "The call record has been transferred to the team: 6 EPOSS-FP." 7 Who were EPOSS-FP? 8 A. I'm sorry, I don't know who that team are. I'm not sure 9 what "FP" stands for. 10 Q. If we go down to the entry after, so 11.48: 11 "The Call record has been transferred to the Team: 12 EPOSS-Dev." 13 Is that your team? 14 A. No. 15 Q. What team is that? 16 A. Well, given that EPOSS is a counter -- well, is 17 an application, I guess it's an applications team that 18 look after -- there were many applications in Horizon, 19 and EPOSS was one of them, so I would assume it's the 20 team who looked after the EPOSS application. 21 Q. Could we go to the next page, please, page 6. There's 22 an entry by Martin McConnell. Who was Martin McConnell? 23 A. I don't recognise that name. 24 Q. He says: 25 "In my first analysis of the message store supplied, 28 1 it would appear that the declarations being written away 2 were done so at the time that the EOD process kicked in. 3 The message which indicates the Riposte failure ..." 4 It says there "putpersistentobject": 5 "... should have allowed the user at least to have 6 backed out and start again, which seems to happen 7 satisfactorily when these conditions are simulated on 8 a development system. As Les has indicated earlier, 9 a system restart should be sufficient to get them back 10 and working. 11 "OK, in which case I would suspect this call should 12 be dropped to a 'B'. Will see if I can simulate the 13 failure whilst in the midst of an EOD scenario." 14 So is Mr McConnell there -- is a fair interpretation 15 of that that he is going to try and simulate what the 16 problem was. Is that a typical response? 17 A. Yes, that's my reading of it. 18 Q. We see there there's a customer call again: 19 "Paged Mike again as per his last request as gone 20 3 pm and call still not resolved. Awaiting his call 21 back to advise." 22 Customer call: 23 "Mike called to advise that if call not resolved by 24 [6.00 pm] then to page the Duty Manager again. 25 "Call updated as requested." 29 1 Then it's the next entry that is really the 2 significant entry on this PinICL that I want to ask you 3 about. Mr McConnell says: 4 "I have talked to Brian Orzel ..." 5 Who is Brian Orzel? 6 A. Brian Orzel was one of our developers and he is also the 7 person who spent quite a bit of time in Escher 8 facilities in the States in a tactical liaison role as 9 well. 10 Q. Spoken to him "about the 'lock' errors written away by 11 Riposte and it would appear that this is an indication 12 of Riposte being rather sick." 13 Is that a technical term? What would you understand 14 by "sick"? 15 A. I'm not sure how to interpret that. There's many 16 possible interpretations. 17 Q. "There are several DIIs ..." 18 What are DIIs? 19 A. I think that's referring to DLLs. 20 Q. DLLs? 21 A. So where that -- so "DLL" and "executable" are computer 22 code. 23 Q. So: 24 "There are several DLLs and executables all being 25 told to go away because of this locking problem. Either 30 1 some application has left some write lock on 2 inadvertently or Riposte is sick as described." 3 Again, "sick", does that assist you at all? 4 A. Again, it's hard for me to interpret what that means, 5 but ... 6 Q. "A reboot should sort this out or try redeclaring on 7 an alternative system. Brian Orzel has suggested 8 routing this for the attention of Mark Jarosz." 9 What do you have to say about the suggestion that it 10 should be for your attention to deal with that issue? 11 A. So I assume from that that Brian wants me to find out 12 from Escher what the right course of action is for this 13 particular error message. What I can't tell from the 14 date was whether Brian was already out there or not, 15 onsite with Escher. 16 Q. Can we look at the first page of this document, please. 17 At the bottom of the first page this PinICL seems to 18 have been sent to Gareth Jenkins on 3 November. What 19 was Gareth Jenkins' role here? 20 A. So within the team, Gareth was the Riposte technical 21 design authority. 22 Q. If we look at the top email, please, Gareth Jenkins is 23 emailing you, presumably following up from Mr Orzel's 24 comment, and he says there: 25 "I don't know if you have been phoned about this 31 1 one. It seems to have been passed to you on the 2 Escher-dev stack." 3 What was the Escher-Dev stack? 4 A. So within PinICL, there's multiple groupings for 5 different people and I think Escher-Dev is one of those 6 groupings. 7 Q. It refers there to what the problem is, including the 8 message: 9 "Timeout occurred waiting for lock." 10 He says: 11 "I assume the problem is down to the previous Query 12 from EPOSS, however I can't see why that would cause 13 a one-off problem on this system. 14 "I don't know if it is relevant, but the machine 15 appears to have been rebooted in the middle of the night 16 a couple of days earlier (ie at 02.00 and twice at 03.00 17 on [30 October]). The counter appears to be at CI4 ..." 18 Now, we mentioned that earlier. We have previously 19 in this Inquiry seen an email to Gareth Jenkins, where 20 Gareth Jenkins is copied in, about CI4 and that email 21 expressed concerns regarding counter performance and 22 code regression with CI4. Is that something you 23 remember at all? 24 A. No. 25 Q. What is Gareth Jenkins asking you to do here? 32 1 A. So I -- well, I think the first thing that he is asking 2 is for confirming with Escher, if this has not already 3 been done previously, what "error 82" means and what the 4 consequences are. 5 Q. Presumably you would have read the PinICL that was 6 forwarded to you. So, at the bottom of this email, he 7 is forwarding the full message to you. Would you have 8 read that at the time? 9 A. I would expect to, yes. I can't remember that 10 particular email but, in general, yes. 11 Q. I mean, those comments about Riposte being "rather 12 sick", that message went to you at least, didn't it? 13 A. Yes, it did. 14 Q. We started today, the first document we looked at, or 15 the second document we looked at was about problems 16 earlier that year with Riposte and you mentioned one of 17 them was resolved but there were two critical issues 18 with Riposte that were mentioned in that earlier 19 document that I took you to. Was this building on your 20 knowledge of issues with Riposte at all? 21 A. So I think the first part of the question is about the 22 error message and what I cannot recollect is whether 23 I have asked this question of Escher before or not, or 24 whether it had to be asked for the first time, about 25 what that error message actually means. So I think 33 1 that's certainly one thing that's being asked in the 2 email. 3 Q. Would you have been concerned to have received a PinICL 4 that said that Riposte was sick? 5 A. Well, in general, yes, and it -- I think the PinICL -- 6 in general with problems like this, unless the error 7 message explains the problem, there is a need to 8 reproduce the problem. So if that's, indeed, what 9 happened, then that would be the right course of action. 10 Q. Was it something that you think should have had Escher's 11 urgent attention? 12 A. Yes, most definitely, based on the priority, yes. 13 Q. Can we look at FUJ00083548, please. Now, on the second 14 page we see the PinICL, it starts on the very bottom of 15 the first page, but it's the second page and it's 16 a PinICL that is from 9 November, so just a week later. 17 The reference here for this PinICL is PC0057478, and we 18 see on the second page, about halfway down, the entry at 19 21.55, it says a critical error was registered: 20 "An error occurred while attempting to destroy 21 a checkpoint run. Timeout occurred waiting for lock ... 22 no suitable kel." 23 Are you able to help us with that at all? It's not 24 listed on that document that I showed you -- the first 25 document that I showed you to identify the relevant 34 1 PinICLs or PEAKs for the Callendar Square problem, but 2 is that also a Riposte lock issue that's being reported 3 there? 4 A. Yes, it is. So the -- this is another example where 5 there's an error message reported by Riposte and the -- 6 whilst I don't recollect this particular example, what 7 I would have done, in general, is I would have taken 8 this to Escher and asked them for feedback about what 9 the error means, what the consequences are on the 10 message store and what the right course of action would 11 be. 12 Q. Would you have taken them to Escher on every occasion? 13 A. Only when asked because I wasn't the only person who was 14 liaising with Escher. So, if I was asked, either by 15 email or verbally, to follow up, then I would do that. 16 I would take the opportunity whilst I was out there to 17 do that. 18 Q. So every occasion you were asked, you would go to Escher 19 and try and resolve the issue? 20 A. Well, I would certainly take the issue to Escher and 21 feed back on the question I was asked. It wasn't always 22 possible in a timely manner because, sometimes when 23 I was working there, the people who I needed to ask 24 weren't there. 25 Q. Can we look at the first page, please. If we look at 35 1 the top -- well, at the bottom it seems, again, to be 2 a PinICL that went to Gareth Jenkins, on 20 November in 3 this case. He emails you at the top on 21 November. 4 They are American date formats but I'm confident that 5 that is 21 November. Why would Gareth Jenkins have 6 emailed you on this occasion? 7 A. Because he wants Escher to confirm details of what 8 "error 94" means. 9 Q. Can you just have a look at this document and tell us in 10 simple terms what's going on. 11 A. So in the third paragraph, starting "However I am 12 curious", he is asking -- he is quoting some error 13 messages that were logged by Riposte and he is then 14 stating he assumes they are benign "but would appreciate 15 confirmation from [myself] before closing the PinICL", 16 and the only way I can seek that confirmation is by 17 asking Escher. 18 Q. Assuming it is "benign", that's something we will see 19 again, is that an assumption that something is going to 20 be okay but it's not a definitive position? 21 A. Well, the -- it's probably building on -- so 22 understanding what the error message means is part of 23 analysing the possible problem it could cause and 24 I think only on conclusion -- once analysis is complete, 25 it could be concluded, maybe, that these messages can be 36 1 ignored. However, I would say, in general, that if it 2 is an error message it does need to be analysed. 3 Q. So again, it's a PinICL, the detail of which is being 4 sent to you by Gareth Jenkins for you to take up with 5 Riposte, is it? 6 A. So my response to this email would be to ask Escher for 7 details of the error message, under what circumstances 8 it occurs and what the consequences are, and then feed 9 that back to Gareth, either verbally, face-to-face or 10 via email, whichever. 11 Q. You would do that in every case when you are asked to? 12 A. Well, where it's a very specific question, "What is this 13 error message?" yes, I would, but if I was unable to 14 have that conversation with Andrew Sutherland then it 15 may be quite a few weeks before there's any response. 16 Q. That final substantive paragraph talks about ClearDesk. 17 Now, I think ClearDesk was a way of resolving this 18 Riposte lock issue because ClearDesk, I think, 19 effectively restarted the system; do you remember that 20 at all? 21 A. I recognise the term "ClearDesk", but I wasn't really 22 aware of the counter architecture and what processes ran 23 when on the counter. 24 Q. Gareth Jenkins says there: 25 "Each time it is put out as part of the ClearDesk 37 1 close down function. ClearDesk continues OK, so again 2 it isn't serious, but we need to avoid any errors being 3 generated at the counter as part of ClearDesk (since 4 they cost Pathway 3p each for a phone call!)." 5 Can you tell us about that, please? 6 A. Yes, so, at this time, the networking was the ISDN 7 dial-on-demand network and what that meant was that 8 there was no connection between the counter and data 9 centre normally but, when there was a need for 10 communication, this ISDN phone call would be 11 established. And what Gareth is asserting there is that 12 if a -- one of the conditions for actually forwarding 13 messages to the data centre -- in this case, it was 14 a Tivoli function -- if there's a red -- an error event 15 logged by anything on the counter, then Tivoli will 16 forward that to the data centre for investigation and 17 that is the phone call that's being referred to. 18 Q. So is that him saying "We would rather not spend the 19 money on the phone calls"? 20 A. Well, given that there were quite a lot of phone calls 21 going on anyway, I'm not sure he is directly concerned 22 about the cost of a phone call because -- I mean, what 23 I would say is that there may well be other reasons why 24 there's a call made anyway at that time. 25 Q. If we look at the very final sentence there he says: 38 1 "... assign the PinICL to me on Escher-Dev until 2 I get feedback from you both." 3 I asked about Escher-Dev before, does this assist 4 your memory, is Gareth Jenkins part of that Escher-Dev 5 team? 6 A. So within PinICL, there's multiple groups and, by 7 implication, Gareth is part of that Escher-Dev team 8 because he's just -- what you said, yes. 9 Q. Can we look at FUJ00083568, please. This is an email to 10 you a few days later, 24 December 2000, and can we look 11 over the page, please, page 2 -- in fact, actually 12 I think we can stay with page 1. 13 The PinICL there, the reference is PC0057957 and 14 that is dated 16 November but it relates to the first 15 PinICL that I took you to, ending 56922, and it says 16 that at the very top of the page. It says "This PinICL 17 is related to" that PinICL, which is the one that's 18 later linked to Callendar Square. 19 Can we look, please, over the page to page 2. 20 Again, it refers to a critical event was registered and 21 it says: 22 "Timeout occurred waiting for lock." 23 So, again, that seems as though it's one of those 24 Riposte lock issues. 25 A. Yes. 39 1 Q. Can we go to page 3, please. 23 November, 11.10: 2 "This event was reported in PC0056922, this call has 3 been closed but the comments from Mark Jarosz, were that 4 if calls of this nature were [over] 1 per month then 5 further investigation should be carried out. In this 6 case I presume that archiving was processing and there 7 was still an outstanding lock on the run table. 8 I presume that the reload of Riposte at ClearDesk will 9 release the locks. Investigating frequency of event in 10 the estate." 11 Now, the suggestion there is that it wasn't on every 12 occasion that you were asked that you would investigate, 13 you would apply some sort of minimum threshold of 14 a problem before going to Riposte. 15 A. So in the example of the error message, then it's very 16 clear that, because within Pathway we didn't know 17 what -- we had no documentation to tell us what the 18 error message meant, we had to ask Escher what it meant. 19 Q. But if you received a one-off incident, or what you 20 considered to be a one-off incident, would you go to 21 Escher? 22 A. If Gareth asked me to, yes, I would. 23 Q. So the suggestion there that really they need to be 24 looking out for a more common occurrence, where would 25 they've got that idea from? 40 1 A. So I think in one of the examples where we discussed 2 reproducing the problem, then the -- that's what we 3 talked about, the frequency of it occurring. 4 Q. We will talk about reproducing in a moment because it 5 seems as though you had concerns that, if it something 6 couldn't be reproduced, there wasn't really any point in 7 going to Escher; is that right? 8 A. Well, it's more a case of if we need to go to Escher 9 because we have found a bug in Riposte -- and this 10 occurs -- this is a more general statement. If we need 11 to investigate a bug then we are very keen to reproduce 12 it so we can then both investigate it with a vendor and 13 also confirm the fixes worked. 14 Q. Sticking with this particular issue, can we go down 15 slightly to the next substantive entry. It says: 16 "This event has some 129 counters reporting this and 17 also MBOCOR02 and MBOCOR03 has reported this event 18 although it may be expected on the Corr servers." 19 So is that correspondence servers? 20 A. I think it is, yes. 21 Q. "I think this needs investigating. Please state what 22 evidence is required will attach Event log/message store 23 & audit logs for this outlet." 24 Then if we go down a little further it says that it 25 is 13.17: 41 1 "The Call record has been assigned to the Team 2 Member: Gareth Jenkins." 3 Then if we look at the first page, it is Gareth 4 Jenkins emailing you. So, again, he has emailed you 5 with the full detail of that PinICL. Would you have 6 read that PinICL at the time? 7 A. I can't recall reading that particular one, but I, in 8 general, would try to keep up with the emails, yes. 9 Q. So the message, for example, that the event has some 129 10 counters reporting, that was sent to you and typically 11 you would read those messages that were sent to you? 12 A. Yes. 13 Q. Now, it says at the bottom there: 14 "I've assigned the PinICL to you on Escher-Dev." 15 Again, so does that assist you with Escher-Dev? 16 A. I'm aware what Escher-Dev is -- 17 Q. Yes. 18 A. -- (unclear), yes. 19 Q. So you were being assigned because you were part of that 20 team? 21 A. Well, I think it was assigned to me because, in terms of 22 it -- the next step in that PinICL, what Gareth was 23 asking for was a definitive statement from Drew on that 24 error message. So the next stage in the workflow for 25 that PinICL would be to update it with that statement. 42 1 Q. And Drew is -- 2 A. So Andrew Sutherland, he is the chief architect from 3 Escher group, the expert on Riposte messaging product. 4 Q. Okay. Mr Jenkins says: 5 "As the PinICL says, this seems to be happening 6 fairly frequently. As far as I can tell, the 7 application is carrying on OK in this case. Since the 8 failure is at midnight, then Riposte is likely to be 9 reloaded fairly soon. 10 "I think we do need a definitive statement from Drew 11 as to whether this event is benign, or what problems we 12 could have when it happens. Could it be due to 13 an application error? Do we need to get more info on 14 when these problems occur. It is clear that the 15 circumstances in this case are very different from those 16 in the original PinICL." 17 Now, Mr Jenkins there seems to be concerned about 18 repeated errors and where they come from; do you agree 19 with that? 20 A. Most definitely, yes. 21 Q. He says there he doesn't seem sure that it's benign, by 22 that stage. 23 A. Well, until we get -- we need the feedback from Escher 24 to explain the error message, which I think we actually 25 got maybe in this example. I don't know if there's 43 1 an email from me with a feedback from Escher. 2 Q. Well, we will come to an email from you. 3 A. Okay, okay. 4 Q. You are being sent that by Gareth Jenkins, again, to 5 take forward with Escher, to take forward with Drew, to 6 see if it's benign or not? 7 A. Yes, although I wouldn't actually ask Drew if it's 8 benign or not, just ask him to explain it and what the 9 consequences are. 10 Q. Can we look at FUJ00083574, please. This is an email 11 from you to Gareth Jenkins. It is about the same PinICL 12 ending 957, and you say there: 13 "Gareth, 14 "From your description it sounds as though we 15 potentially have a recipe for a reproducible case. 16 "I will try this today and also in parallel chase 17 Drew for a response on what this event means and whether 18 we should be concerned." 19 The reference there to "reproducible case" -- 20 A. Yes. 21 Q. -- again, I think we discussed this briefly, but it's 22 something that does crop up from time to time, and it 23 looks like what is being said is that, without 24 a reproducible case, it's difficult to progress the 25 problem. Is that something you agree with or not? 44 1 A. Yes, it's much more difficult to progress a problem that 2 we can't reproduce, yes, unless it's a previously known 3 issue. 4 Q. It looks from this and other correspondence that you do, 5 at least, apply some criteria in respect of following 6 things up with Escher. In fact, in this case, you say 7 you are going to, but if it wasn't a reproducible case, 8 if it seemed like a one-off issue, would you always send 9 it to Escher? 10 A. Yes, most definitely. The reason for mentioning the 11 potentially reproducible case, is that it makes the 12 interaction with Escher potentially much more productive 13 because, as well as asking them what could happen, we 14 can actually demonstrate what is happening. 15 Q. Can we go to FUJ00083582, please? This is now 16 1 December 2000, and this is -- is this an update to 17 Gareth Jenkins on this issue? 18 A. So this -- in this case, I have responded to his 19 question about the particular error message and what the 20 feedback I had from Drew was, as -- 21 Q. Sorry. You say there: 22 "Hi Gareth, 23 "I can confirm (having checked with Drew) that 24 a timeout of this sort is likely to be benign in the 25 sense that it should not result in a message store 45 1 corruption. 2 "However had the operation which was affected by 3 this timeout been a message server internal operation, 4 for example and index maintenance thread operation, then 5 an additional error ... should have been logged. 6 "Therefore a possibility is that an API call has 7 timed out and the application is not checking for error 8 events." 9 Now, that update: likely to be benign, "should not 10 result", possibly an API call has timed out, et cetera; 11 would you accept that those are quite caveated 12 responses? 13 A. Yes, they are, based on conversations with Escher and 14 the limited information we have available, trying to say 15 what could be happening. For example, Escher are making 16 the point that, if something was affected in the message 17 server, there would have been further error messages 18 and, as it's their product, they can say that's by 19 design. So, even though I used the term "should have 20 been logged", I maybe should have used the term "highly 21 likely" that it would have been logged, because Escher 22 said so. 23 Q. But it looks from that message that you haven't got to 24 the bottom of the problem? 25 A. That is definitely the case, yes, because the next part 46 1 is very significant and this, again, is based on 2 conversations with Escher, that because there's an error 3 message and something has timed out, then something was 4 trying to happen, and if it wasn't an internal message 5 server operation, because Escher said so, then the 6 suggestion is that the -- and we know there was an agent 7 because Gareth mentioned this running at the time, then 8 the agent may have caused -- an agent operation may have 9 caused the error, which is why the suggestion from 10 Escher was "Check that the agent is validating all 11 responses from interactions with the Riposte message 12 server". 13 Q. I will come to that (a) and (b) in a second, but the 14 words, for example, "mostly benign" or "relatively 15 benign" are words that we have seen elsewhere and we may 16 see in further emails, and again "likely", "should", 17 et cetera. Does that indicate, perhaps, that you 18 couldn't be sure that there wouldn't be serious problems 19 arising from this Riposte lock issue at that stage? 20 A. So I'm definitely not sure that is the case and there is 21 further investigation needed, yes. 22 Q. Then looking at those (a) and (b): 23 "In terms of progressing ... I would suggest ..." 24 So this is you suggesting, not just simply passing 25 a message, but you are coming up with your own solution? 47 1 A. No, that was not the case. When I discussed this with 2 Drew and we made the observation that there were no 3 other error messages from the message server, he stated 4 that, as there was an agent running, then the agent 5 possibly would have had error responses, which should 6 have been logged and possibly they weren't, which is why 7 the recommendation for (a) is directly as a consequence 8 of what Drew asked me to do: is the agent checking all 9 the responses correctly? 10 Q. But you weren't simply passing a message, you were 11 applying your own mind to the issue as well, weren't 12 you? 13 A. So in the case of (a), it was Drew's recommendation to 14 check that the application -- the agent or the 15 application using the Riposte message server was 16 checking its responses correctly. 17 Q. No, I mean, let's look at it. (a) says: 18 "Get the LFS Agent code checked to confirm that all 19 API calls have error checking. I am happy to do this if 20 the developers are prepared to send me the source." 21 Now, we have heard about issues accessing Escher 22 code. Is that referring to an issue accessing the 23 original code? 24 A. No, this is -- the LFS agent code is our code, so the -- 25 Q. So who were the developers that you are talking about 48 1 there? 2 A. Well, the development team that created that agent. 3 I don't know who the individuals are. 4 Q. So you're saying there that you're happy to assist if 5 the ICL Pathway developers are prepared to send you the 6 code? 7 A. Yes, I mean, what I should have said -- it's a bit 8 tongue in cheek. I should have said "They should check 9 it themselves", because they should be checking all 10 replies. 11 Q. Were there issues internally with getting hold of source 12 code? 13 A. No, the -- because these were internal people doing 14 development, they would have the source code for their 15 own agent. So they could get this checked relatively 16 easily. 17 Q. "(b) Continue to try and reproduce this problem. 18 Knowing what the Agent is doing (either source code or 19 some design documentation) would be useful." 20 So it seems there that a solution is to keep on 21 trying to reproduce it. So, at that stage, it seems it 22 hasn't been reproduced or is not yet reproducible? 23 A. Yes. 24 MR BLAKE: Just looking at the time, sir, shall we take 25 a short break now? I probably only have 20 minutes left 49 1 and then there will be some questions from recognised 2 legal representatives. We can either take a break now 3 or in 20 minutes. 4 SIR WYN WILLIAMS: No, let's do it now because I think we 5 have to think of the transcriber as well. So let's do 6 it now. 7 During this short break, Mr Jarosz, would you please 8 not speak about your evidence with anyone. All right? 9 A. Okay. 10 SIR WYN WILLIAMS: Thank you. 11 MR BLAKE: Shall we say 25 past or half past? 12 SIR WYN WILLIAMS: If the people in the room think that we 13 will complete the witness comfortably before lunch, I'm 14 happy with half past. 15 MR BLAKE: Yes, I think the answer is, yes. 16 SIR WYN WILLIAMS: Right, half past then. 17 (11.16 am) 18 (Short Break) 19 (11.30 am) 20 MR BLAKE: Hello, Chair, we can see you. Can you see and 21 hear us? 22 SIR WYN WILLIAMS: Yes, I can, thank you. 23 MR BLAKE: Thank you very much. 24 Before the break, the first documents I took you to 25 we saw early Riposte issues in the summer of 2000 and 50 1 then I took you to some PinICLs that were later in 2000, 2 that addressed things such as the Riposte lock. 3 Can I take you to POL00028911, please. Now, if you 4 would like to take some time over this document, please 5 do. It's a document that we're not too sure where it's 6 come from. It may have been written by Gareth Jenkins, 7 that may be established in due course. It's a document 8 that I took you to first today, and I showed you the 9 PC0056922 being referenced there. 10 Can we scroll down this page, please, and actually 11 over to the next page. There's some analysis there, 12 analysis of PEAKs, and it said: 13 "They are all related to different incidents of the 14 same fundamental error message from Riposte." 15 Then, "How we dealt with the problem", it says: 16 "When first spotted in 2000, an avoidance action was 17 identified and this was identified in the KEL. The 18 advice was for SMC to monitor the associated events and 19 then alert the branch. It isn't clear how effective 20 this was." 21 Then it says: 22 "Analysis of PEAKs quoted. Which of them truly 23 refer to same issue. 24 "They all relate to the same Riposte error. It 25 isn't clear why this re-occurred in 2010 after the 51 1 Riposte fix in 2006." 2 Then there's a section on "Scope". It says: 3 "The root cause of all these was a bug in Riposte 4 that had the effect of preventing a counter from writing 5 messages -- either those being replicated to it or those 6 generated on that counter. 7 "This was not always immediately obvious to the user 8 of the counter. This could result in them thinking that 9 some transactions which had been entered, were missing, 10 and so they attempted to re-enter the transactions on 11 another counter. When the offending counter was 12 re-started, both versions of the transaction became 13 visible and this could cause errors in the accounts. 14 "Attempting to balance the branch when a counter was 15 in this state could also result in errors." 16 Is that something that you remember? I'm not 17 talking about this particular document but that kind of 18 a summary of the problem? 19 A. No, I've never seen this before. It's the first time 20 I have read it. 21 Q. But the issue, does that accurately describe for you the 22 Riposte lock problem, or the consequences of the Riposte 23 lock problem? 24 (Pause) 25 A. So the first paragraph there about the root cause -- 52 1 Q. Yes. 2 A. -- the analysis conducted, then I can see how that could 3 be a consequence of the Riposte lock problem and, given 4 that someone has done that analysis, it makes sense to 5 me, yes. 6 Q. The reason I'm taking you to this document now is that 7 it addresses some of the things that you said this 8 morning, and I just want to turn over the page, please. 9 There is some analysis of those PinICLs from 2000 and it 10 is that first substantive paragraph, and I'm going to 11 read it for the record. It says: 12 "However, on re-reading PEAK PC0126376, I can see it 13 refers to 2 KELs (which I presumably didn't look at back 14 in 2010), which were raised much earlier. This shows 15 that the Riposte issue had been initially identified 16 back in 2000. This is made clear in KEL 17 JBallantyne5245K and the associated PEAK PC0056922. 18 This shows that there is a problem in Riposte such that 19 if it loses a Thread which holds a critical lock, then 20 Riposte grinds to a halt and the counter becomes 21 [unstable]. The avoidance action is to restart the 22 [computer]." 23 Just pausing there, do you remember advice being 24 given to avoid it by restarting the counter? That's 25 something we addressed this morning, I just wanted to 53 1 know if that jogged your memory at all? 2 A. I remember you mentioning about that being stated, but 3 it's -- it's not advice I would ever give or agree with. 4 Q. But it was mentioned in the PinICL that you received? 5 A. Yes. 6 Q. "The symptoms of the problem are a large number of 7 events. The PEAK advises that if the issue occurs more 8 than once per month, then we would need to try and 9 reproduce the issue. The KEL also refers to PC0083101. 10 "Past experience shows that Escher wouldn't consider 11 bugs if they are not reproducible." 12 Now, that's something I asked you about this 13 morning. Do you think that that statement is right or 14 wrong? 15 A. So my take on that statement is that, if the bug isn't 16 reproducible, then it makes progressing the root cause 17 analysis much, much more difficult. But I'm aware 18 that -- or, on at least one occasion, when there was 19 a bug, potential bug in the message server, Andrew 20 Sutherland came to Bracknell to investigate it. So 21 there's an example, I think, where we couldn't send him 22 a reproducible case but he attended the facility in 23 Bracknell to investigate. 24 Q. Do you think that it was common knowledge amongst those 25 who worked on these issues that it wouldn't really be 54 1 worth troubling Escher, and perhaps not troubling you, 2 if it was a case of a bug that wasn't reproducible? 3 A. Well, I think, where -- I mean, the objective is to 4 understand the issue and to close it and, in the case 5 where that can be done, based on existing evidence, then 6 that could be relatively straightforward. However, in 7 many cases, a lot of effort needs to be expended in 8 reproducing the problem to investigate it further and 9 I can think of a number of occasions when we had to do 10 that, so I don't think -- if a problem warrants 11 investigation, then it needs to be investigated, and 12 just because it's difficult to investigate it, isn't 13 a reason not to investigate it. 14 Q. Might it sometimes be called a once-off error if it 15 couldn't be reproduced? 16 A. Well, if it only ever happens once, and it can't be 17 reproduced then, yes, it could be labelled as it only 18 happened once, yes. 19 Q. Very briefly, it says: 20 "The PEAK was then closed and the KEL 21 JBallantyne5245K produced. In particular the KEL 22 advises SMC (who monitor events from counters), that if 23 such events are seen to phone the branch and advise them 24 to restart the affected counter, and if they are 25 balancing to abandon the balance until the reboot has 55 1 happened as this prevents replication working 2 correctly." 3 We don't need to spend any more time on this 4 particular document. We can ask those who are familiar 5 with this document about the document itself. 6 I want to move on to 2001 and can we look at 7 FUJ00083592, please. So we're now in 2001 and can we go 8 over the page. This is an email from Brian Orzel who 9 you mentioned earlier. It's to a limited number of 10 people: David Richardson, Chris Wannell, yourself, 11 Gareth Jenkins, Lionel Higman; who are those people? 12 A. I recognise the names, but I can't remember their roles. 13 Q. Is there any significance after Gareth Jenkins' name is 14 says "GL" or "GI", could those be initials, perhaps? 15 A. I think they're initials in the email address. 16 Q. This email says: 17 "Gents, 18 "It will take a little time for the new 'users' to 19 bed in." 20 Do you know who he is talking about there? 21 A. No. 22 Q. "I am not actively working on anything in the 23 '[Control-inbox]' or 'Parked'. If you have a pet PinICL 24 therein that you think I should be chasing then come 25 over and beat me up." 56 1 He lists below a large number of PinICLs and I think 2 there's one -- well, can you help me? If we scroll down 3 we can see that there are some that are parked, they 4 have various names on. Why would you be sent this? 5 A. I think there's a -- because -- the only reason I think 6 I would be sent this is if there are some PinICLs that 7 are assigned to me. 8 Q. Yes. Let's go to page 1., it may assist us. If we look 9 at the bottom there, it's an email from Gareth Jenkins. 10 Again, Gareth Jenkins directly to you: 11 "Mark, 12 "Please can you have a look through the 7 PinICLs in 13 the list assigned to you. I suspect that many of them 14 can either be closed or 'Parked'. I can supply you with 15 more details about them if you have problems in getting 16 through to PinICL." 17 What was Gareth Jenkins' role here? 18 A. I think he is just pointing out that some of the PinICLs 19 are assigned to me and that they have -- I assume that 20 they have been open for a while and need to be 21 concluded. 22 Q. You are one of the original recipients of the email that 23 he is replying to on, or forwarding to you. You would 24 have seen the original email. Why would Gareth Jenkins 25 particularly be asking you there about seven PinICLs in 57 1 the list assigned to you? What was his role in relation 2 to your role there? 3 A. I can't think why he would be asking me to do this 4 because he -- no, I can't think of a reason. 5 Q. If we look over the page and look at that list, there 6 are quite a lot that say "At-Escher". Now, would it be 7 right to say that they couldn't be addressed by Fujitsu 8 because they were reliant on Escher to provide the 9 solution in some or all of those cases? 10 A. So the -- I guess the important thing is that quite 11 a bit of the Code used in our solution did come from 12 Escher. So, in those cases, they would have to -- they 13 were quite rightly -- if there's a problem with the 14 code, they would need to resolve it. 15 Q. Were you aware of issues obtaining code from Escher? We 16 have heard about difficulties in obtaining the original 17 code because of intellectual property reasons or -- 18 A. Yes, I wasn't referring to source code. I was referring 19 to applications. So, for example, what Escher provided 20 us was the message server, the -- at one time, there was 21 a counter application they provided and they also 22 provided the -- the overarching application that ran on 23 the counter, known as the desktop. So, if we identified 24 in our testing, problems in those areas, then the right 25 place for it to be investigated would be with Escher. 58 1 Q. Now, we have quite a few "At-Escher" and we also have 2 some that are duplicates, I think, and also some that 3 say "Parked"; is that right? 4 (Pause) 5 So we're there onto some "Duplicates" -- 6 A. Yes. 7 Q. -- and then, if we keep on scrolling, I think there are 8 quite a few that are parked as well. Yes? 9 A. Oh, sorry, yes, I have seen both "Parked" and 10 "Duplicates", yes. 11 Q. Might some of those ones that were parked have been 12 parked because they couldn't be reliably reproduced at 13 that time? 14 A. I'm not sure of the criteria for going into a parked 15 status, as opposed to open. I didn't use PinICL as part 16 of my kind of daily workflow. So I don't know what the 17 kind of workflow rules were for it. 18 Q. In relation to Gareth Jenkins -- so if we go to the 19 first page -- is a fair description of this email that's 20 been sent to him an email that contains a list of 21 outstanding bugs, errors and defects with Horizon? 22 A. So the email looks to me to be a summary of PinICLs 23 which are, I guess, in an open state, ie they haven't 24 been closed, and the -- in terms of what they're 25 referring to, there could be a combination of bugs or, 59 1 you know, seeking information. It's hard just looking 2 at the title to categorise what they fall into. 3 Q. Perhaps a significant list of incidents being sent to 4 Gareth Jenkins in 2001, would you agree with that? 5 A. Well, given that the purpose of the system was to -- 6 well, so there's one example -- it's quite fortunate in 7 this email Chris Wannell is pointing out that there's 8 a PinICL which also refers to an item which is on the 9 RER, which is the Riposte Enhancement Register, so Chris 10 is saying, quite rightly, it shouldn't be a PinICL 11 because it's an enhancement request, as opposed to 12 a design -- as opposed to the Escher code not working as 13 it should. So there's just one example there, I think, 14 of where the PinICL system is being used for something 15 that is probably not really an incident, but I think, in 16 general, yes, the majority would be incidents. 17 Q. Thank you. Can we go to FUJ00083600. Moving now to 18 11 May 2001. Now, this is an email, again, from Gareth 19 Jenkins to yourself, and he says: 20 "I have received this PinICL. 21 "I know I've raised with you before the question of 22 Error 82, though in the past it's been on counters. I'm 23 also aware that the error itself is benign, though it 24 could result in other errors to agents (for example)." 25 It gives some detail there. Again, it refers in 60 1 that detail to "Timeout occurred waiting for lock", so 2 is this, again, a Riposte lock issue? 3 A. Yes, it is. 4 Q. Then if we look at the bottom, final paragraph of this 5 page, Gareth Jenkins says there: 6 "What I'm really asking is for confirmation that the 7 associated errors are indeed benign, in which case I can 8 ensure that KELs are raised so as to suppress the 9 reporting of them in future. It worries me that 10 messages are failing to be inserted, however if they are 11 being replicated, then I guess it doesn't matter!" 12 Do you remember this email at all? 13 A. I didn't remember it until I saw the material earlier on 14 in the week. 15 Q. Gareth Jenkins there is talking about a large number of 16 errors in this particular case and he is worried that 17 they may not be benign. Is that a fair characterisation 18 of that final paragraph? 19 (Pause) 20 A. Well, looking at the error messages he -- for example, 21 part way down the page, the third occurrence was 22 somewhat different, the Riposte error where there's 23 a "RiposteStartTransaction" exception, that's an error 24 that hasn't -- I'm not aware we have asked Escher about 25 that before, so it would need to be followed up with 61 1 them, because it's reporting a problem with a Riposte 2 function. 3 Q. But looking, I mean, for example, at those first ones, 4 it is very clear that some of them relate to the Riposte 5 lock problem "Timeout occurred waiting for lock"? 6 A. Yes. 7 Q. The same error that we have heard a number of times this 8 morning. You knew Gareth Jenkins. Was his concern 9 there genuine? Did you feel it was genuine? Did you 10 feel his general approach to these kinds of issues was 11 one of being worried, for example? 12 A. So I think his concern is genuine and where he is asking 13 for confirmation that the associated errors are indeed 14 benign, I think it would be quite difficult to provide 15 that confirmation, based on what I'm seeing in front of 16 me. 17 Q. He is looking to you for help there, isn't he? 18 A. Well, he is asking me to -- yes, he is, and I would have 19 to ask Escher. I cannot recall asking Escher about that 20 particular message, but I would have to ask them and 21 then provide -- but, in the previous explanation, I did 22 state to Gareth that where Escher confirmed that, from 23 a message store perspective, it's unlikely there was 24 an adverse impact, the -- from an application point of 25 view, it's very important to confirm that the 62 1 application is checking all the return codes. 2 Q. So he was aware of the information you had passed to him 3 earlier? 4 A. Yes. 5 Q. But yet he is still asking, in 2001 -- I think that's 6 May 2001 -- "Can you just really please check whether 7 they are benign?" 8 A. I mean, the thing is, what I can see happening, just 9 under "The 3rd occurrence was somewhat different" 10 section, it states -- that error message states that 11 that particular function failed, therefore -- 12 an application was trying to do something and it failed, 13 so the -- it really depends on what the consequences of 14 that are. 15 So, based on what I see in front of me, I could 16 never confirm that is benign. I would need to ask 17 someone to look into what was happening at the time. 18 That would be the recommendation. 19 Q. I think you said that you don't recall following that 20 up? 21 A. No, not this one. I mean -- I just cannot recall 22 discussing this issue. 23 Q. Let's move to 7 August 2001, FUJ00083608, please. So 24 here we are, August 2001, we have an email to yourself 25 from Gareth Jenkins. I think you are the recipient, 63 1 there are a couple of people copied in there. He sends 2 you an Escher-Dev PinICL stack, those are listed there, 3 and can we look down at the bottom. Many of them seem 4 to relate to Riposte. He says: 5 "I know the last one is assigned to me, but I sent 6 you an email about it in July and am about to reassign 7 it to you. 8 "The current situation on most of them I believe is 9 that they are 'one-off' problems, and perhaps we should 10 consider closing them. If you want help in accessing 11 the PinICLs or their history, then please let me know." 12 Again, I mean, he seems to be asking you for 13 guidance there, isn't he, or assistance at least? 14 A. Yes, he is, because, in general, with the Riposte 15 message server, at that time, we did need to liaise 16 directly with Escher to get advice, so that's what 17 I would be doing. 18 Q. It says there: 19 "... I believe that they are 'one-off' problems ..." 20 Does this go back to the reproducible issue that 21 perhaps they were ones that couldn't be reproduced? 22 A. So I think the use of the term "one-off" applies to how 23 often they are being observed, only once, because there 24 could be a problem which is -- which was happening 25 regularly but it's still difficult to reproduce it in 64 1 a development environment to diagnose it further. 2 Q. Does that rely on somebody connecting all the dots from 3 the one-off incidents though, to work out whether there 4 are common themes? 5 A. Most definitely, yes, it does. A lot of data analysis 6 would be needed. 7 Q. Let's move to 2 May, FUJ00083621. Now we're looking at 8 the bottom of that page, PinICL PC0075892. Again, 9 that's one that's been linked to the Callendar Square 10 issue. Let's look over the page to page 2, and you have 11 the customer call there, 2 May 2002. Can we scroll down 12 a little bit. It says there: 13 "An unexpected error occurred while attempting to 14 insert a message. Timeout occurred waiting for lock." 15 Again, we hear that same phrase: "timeout occurred 16 waiting for lock". 17 Can we go over the page, please, towards the bottom 18 of that page. You have John Simpkins, again, 2 May at 19 4.03 pm: 20 "These events have stopped occurring now and the 21 Tivoli monitoring can be restarted. 22 "The events started at [5.29] on 1 May 2002 after 23 the counter was rebooted. The counter produced one of 24 these messages every 10 seconds throughout the night 25 until ClearDesk restarted Riposte at 03.34. This 65 1 cleared the lock and the system has been fine since." 2 Then over the page, page 4, another substantive 3 entry by John Simpkins: 4 "Appears similar to a problem we had on the 5 correspondence servers some time back where a lock on 6 the check point would kill agents. 7 "Attached application log as evidence. Passing to 8 development for comments." 9 Then we look at page 1 and this is, again, a PinICL 10 that's sent to Gareth Jenkins and, again, it's got 11 Gareth Jenkins asking you follow-up questions. This 12 time we're now in May 2002. Again, Gareth Jenkins seems 13 to be asking you for your opinion. He says: 14 "Any thoughts on this one? Unless there is 15 something obvious to investigate I suggest we will 16 probably need to write this off as a 'one-off'. Is it 17 worth trying to find out why the machine was rebooted?" 18 So he doesn't seem there to be asking you simply to 19 make contact with Riposte. He does seem to be asking 20 you for your substantive opinion on a particular 21 problem, doesn't he? 22 (Pause) 23 A. In this case, I think we would need to confirm what 24 those -- the right course of action would be to seek 25 confirmation from Escher what those error messages mean 66 1 and what the consequences are. 2 Q. Time and time again we have seen emails from Gareth 3 Jenkins to yourself. He is not just asking you to 4 contact Escher and be the message man. I mean, he is 5 really asking you for your thoughts on this particular 6 problem. 7 A. But the only way I could contribute to the conversation 8 with Gareth would be to liaise with Escher because, 9 without any documentation on their message server, the 10 only way I can gain knowledge is by speaking with 11 Escher. 12 Q. He is saying there that he will probably need to write 13 it off as a one-off. Again, I mean, this is a problem 14 with Riposte in the error message. I imagine 15 subpostmasters will be asking how many one-offs makes 16 something not a one-off. 17 A. What isn't in the email is any context about what the 18 application was doing at the time, if anything. 19 Q. This phase is focused on rollout 2000, et cetera. We 20 know that the Callendar Square bug continued until at 21 least 2006. There was an S90 software fix; is that 22 something you're aware of? 23 A. No. 24 Q. It had the potential to cause discrepancies. Shouldn't 25 this Riposte lock issue have been front and centre of 67 1 your witness statement? 2 A. No -- so when I produced the initial witness statement 3 at the time, my recollection of the Riposte errors were 4 as I described: requesting information from Escher as to 5 what they mean and what the consequences could be. 6 Q. The picture that's built up this morning is that you 7 were quite involved in this particular issue, weren't 8 you? 9 A. Even though we have focused on this, it was a very small 10 part of my normal role within the programme. 11 Q. These continued problems with the Riposte lock, do you 12 know if anyone was feeding those problems back to the 13 Post Office? 14 A. I don't know and I don't think I would know. 15 Q. Did you ever -- 16 SIR WYN WILLIAMS: Could the document be taken down, please? 17 MR BLAKE: Did you ever speak to any subpostmasters directly 18 about issues with Riposte? 19 A. No. 20 MR BLAKE: Thank you, sir. Those are all my questions. 21 Mr Jacobs, I think, is first. 22 SIR WYN WILLIAMS: Over to you, Mr Jacobs. 23 Questioned by MR JACOBS 24 MR JACOBS: Thank you, sir. Can I just check that you can 25 see me and that you can hear me? 68 1 SIR WYN WILLIAMS: I can, yes. 2 MR JACOBS: Thank you. 3 Mr Jarosz, I have some questions for you on behalf 4 of 153 subpostmasters who were pursued by Post Office 5 for shortfalls that were apparent which they couldn't 6 check. 7 I want to ask you about replication. In your 8 statement at paragraph 21(d), if we could call that up 9 on the screen, it's WITN04810100, page 10 of 22. Thank 10 you. I'm just waiting for it to come up on the screen. 11 Thank you. 12 So you talk about an approach taken whereby messages 13 were replicated and: 14 "... the system created multiple copies of a message 15 on each message store." 16 Is that right? 17 A. So on each counter -- so on each counter apart -- so on 18 each counter there was a single message store. 19 Q. Yes. 20 A. And if there are two or more counters in a branch then 21 each of those counters would have its own message store 22 and the Riposte behaviour was to -- if a message got 23 created on the third counter, it will be replicated to 24 every other counter in the branch. 25 Q. Right, and I think the position is that, if one counter 69 1 was down, the other counter would "know" the message on 2 the counter that wasn't functioning. 3 A. So in that scenario, if replication is working 4 correctly, then each counter gets a copy of messages 5 from every other counter and also from the 6 correspondence servers in the data centre, so within 7 a given message store, yes, you see messages for every 8 counter and the correspondence service. 9 Q. The reason I have been asked to ask this question is 10 because many of our clients, when they gave evidence in 11 the Inquiry in February to March of this year, came up 12 with quite a similar issue where they would have 13 a shortfall, say for example £2,000, they would go into 14 the system to try and resolve it and it would come up at 15 £4,000, then it would come up as £8,000, and it would 16 keep replicating. 17 The question I have is: could it be the case that 18 these replicated shortfalls arose from the replication 19 system that you have described not working correctly in 20 addition to or alternatively to bugs, errors and defects 21 that we know about? 22 A. So I think I would answer in two parts. The first part 23 is, if the replication wasn't working correctly, then 24 there could be a number of scenarios. For example, some 25 counters would be missing messages from other counters, 70 1 possibly because of a -- the network in the branch was 2 partitioned. So I think a plausible scenario, which 3 I can envisage would be in a multi-counter office, if 4 a network gets partitioned anyway, then some counters 5 won't be able to replicate to other counters. 6 Now, in terms of how that would manifest itself, it 7 would mean that the counters which cannot reach 8 a gateway have no online communication with the data 9 centre. So there might be some observable incident as 10 a result of that. It depends what proportion of 11 transactions were online and what proportion were 12 performed locally. 13 Q. If that did happen, if the system got stuck in this way 14 and there was no connectivity, I think your evidence is 15 that there was something called a gateway node, so that 16 everything would sort of feedback in once it was 17 restored. Is there a possibility, is it plausible, that 18 that part of the process could lead to subpostmasters 19 having their shortfalls doubling up through 20 a malfunction of this part of the system? 21 A. So the special role of the gateway is it is the only 22 counter which communicates with the correspondence 23 servers at the data centre. So in the scenario 24 I described of the network being partitioned, what that 25 would mean is that the gateway and some other counters 71 1 would have a -- would have messages being created and 2 communicating with the data centre, whereas some other 3 counters would be isolated and, therefore, then messages 4 wouldn't be replicated until the network was restored, 5 so there would be different messages in different parts 6 of the network. 7 In terms of the consequences of that on the 8 application, unfortunately I can't -- I have no 9 expertise in that, how the application would interpret 10 that scenario. But, certainly, from a network point of 11 view that could happen and the thing I would mention, of 12 course, is in a single counter office, there's only one 13 counter, it's the gateway counter and, in that case, 14 there's two Riposte message servers on the counter 15 replicating to each other. And the reason for that is, 16 should that counter fail, then it has a removable drive 17 so the replacement one can be initialised from that. 18 Q. So I think what you're saying, and correct me if I am 19 wrong, is that, although you're not able to be 20 absolutely clear, it's possible that the scenario that 21 I have described could have arisen from a malfunction of 22 this part of the system? 23 A. Yes, definitely, because, even though in my witness 24 statement I state how it's designed to work, clearly 25 networks do fail for periods of time and therefore this 72 1 partitioning can occur. 2 Q. Thank you. The next question that I have for you 3 relates to connectivity in remote areas and this is in 4 relation to paragraph 38(b) of your statement, which is, 5 again the same reference, WITN04810100, paragraph 38(b) 6 please, page 17 of 22. We can see towards the bottom of 7 that section you say: 8 "I recall there were about 140 branches where we 9 could not use ISDN as the branches were very remote. In 10 those cases, as ISDN was not available, we used 11 VSAT ..." 12 We know from above that means "very small aperture 13 terminal": 14 "... as an alternate means of connection. VSAT is, 15 effectively, a satellite connection and, as with any 16 network solution, its reliability depends on the context 17 in which it is deployed. For instance, VSAT reliability 18 can be affected by inclement weather." 19 Again, the reason I'm asking this question is 20 because it arises from the experiences of some of our 21 clients, who say that they experienced power outages and 22 shortfalls arose often after there were power outages. 23 Now, what I wanted to ask you is: you said here that 24 VSAT reliability can be affected by inclement weather. 25 What sort of weather conditions would affect that 73 1 reliability? 2 A. Rain and snow, for example, because they attenuate the 3 signal. 4 Q. So this is to do with ... 5 You say that: 6 "... as with any network [position] its reliability 7 depends on the context in which it is deployed." 8 What were the other issues that affected VSAT 9 reliability? 10 A. So as well as the weather conditions, the VSAT service 11 that we used was from a single provider. 12 Q. Yes. 13 A. Slightly different to the ISDN service, where, because 14 it's geographically distributed, there are multiple 15 exchanges being used. So if this provider, for example, 16 has some problem in their network, then it could affect 17 all or multiple branches that relied on VSAT for 18 communications, for the period of time that that problem 19 persisted. 20 Q. Okay. Do you accept because of this, those who were in 21 rural areas were more vulnerable to difficulties with 22 the system than other subpostmasters? 23 A. The -- so I'm trying to think what characteristics would 24 be affected by rural areas, so certainly the ... I'm 25 trying to think of a characteristic of the network which 74 1 was affected by distance from exchange or VSAT. 2 (Pause) 3 I'm struggling to come up with a plausible scenario 4 which would differentiate the network characteristics. 5 There may be one, I just cannot think of one off the top 6 of my head. 7 Q. Well, I will move to my next question. Could 8 an unstable connection affect post office systems or 9 balances? 10 A. Well, so an unstable connection would -- we're talking 11 about the connection from the gateway now, into the data 12 centre -- 13 Q. Yes? 14 A. -- as opposed to within the branch? So it would 15 certainly affect message replication between the branch 16 and the data centre and the -- though it would manifest 17 itself as where either the data centre or the branch 18 need to communicate with each other because they need to 19 exchange messages for some application reason, but they 20 are unable to, or it happened intermittently, so that 21 would certainly happen and, again, the consequences of 22 that on the application obviously depend on the 23 application, but yes. 24 MR JACOBS: Thank you. I'm just going to see if I have 25 anything else to ask. 75 1 That covers all the questions I have. Thank you 2 very much. 3 A. Thank you. 4 SIR WYN WILLIAMS: Do we have any other questions? 5 MS PAGE: Yes, sir, some questions from me please, sir. 6 SIR WYN WILLIAMS: Very well. 7 Questioned by MS PAGE 8 MS PAGE: I'm Flora Page. I'm also acting for a number of 9 the subpostmaster Core Participants and I'm also going 10 to focus on what I understand to have been your 11 responsibility, which was the network solution, and 12 your -- that means, doesn't it, that you were 13 responsible for the design of the counters communicating 14 with the central data hubs; is that right? 15 A. Yes, for the network service that we provided to enable 16 that communication to take place. 17 Q. Have you had a chance to look at a section of the report 18 from Mr Charles Cipione, which he headed with the title 19 "Many Post Office branches were disconnected from the 20 central system during national rollout"? Does that ring 21 a bell at all? We can bring it up. 22 A. No, it doesn't, but -- I mean, what I would say is in 23 general that branches being disconnected from the 24 central system would happen when -- for example, if it 25 was an ISDN outage, which is why we had other solutions 76 1 in place to deal with that. 2 Q. Well, let's just, if we can, we will bring up 3 EXPG0000001, and this is Mr Cipione's report. If we 4 look at page 83, please. 5 So we see that heading there, it takes perhaps 6 a little bit of unpacking but he talks about how the 7 design used -- in that second paragraph he talks about 8 the design feature was a telecommunications system which 9 depended on ISDN or, in some cases, satellite links and 10 I think that ties up with what you have already told us, 11 doesn't it? 12 A. Yes. 13 Q. It says at 10.1.3 that: 14 "The Monthly Reports indicate throughout 1998 and 15 1999 that ICL Pathway was concerned with their ability 16 to effectuate this design feature: they were concerned 17 with BT's coverage of the UK as well as other technical 18 issues related to their standards." 19 Then it says, in the following paragraph, 10.1.4: 20 "During the national rollout these problems were 21 realised. Hardware, network availability and user 22 issues combined to create a situation where ICL Pathway 23 was occupied with a higher than expected amount of 24 non-polling branches." 25 He explains there are two problems associated with 77 1 that: 2 "This was problematic because [it] relied on the 3 telecommunication design aspect ... to collate and 4 centralise information on all the activity of the 5 branches, but also to allow for efficient updates of 6 software to the branches." 7 Does that make sense to you? 8 A. It does, yes. 9 Q. All right, so the "polling", that's just a terminology 10 for the branches connecting to the central servers, 11 isn't it? 12 A. Yes. 13 Q. He then goes on a bit further on in this section to 14 provide statistics on the numbers of branches which were 15 not polling or didn't poll for significant periods of 16 time. He has already identified there, hasn't he, the 17 issues that result from that: the former one being the 18 data not actually managing up, so things not getting to 19 the central data, which should have done, from the 20 counters; is that fair? 21 A. Yes, if they were disconnected then that would happen, 22 yes. 23 Q. Were you conscious at the time of rollout, and surely 24 you should have been, that non-polling was an issue? 25 A. I wasn't conscious that there was a higher -- I wasn't 78 1 conscious that it was a higher than expected amount of 2 non-polling branches but non-polling was a consequence 3 of the network solution because there was no resilient 4 networking -- at this point in time, so I'm thinking in 5 this period of time up to 2000, there was no network 6 resilience for branches, so if the primary network 7 service wasn't functioning, then there would be 8 non-polling. 9 This was one of the reasons for introducing the 10 manual back up process. 11 Q. When was that introduced? 12 A. I'm not sure when that was deployed, but this was the 13 process when an engineer would go to the branch and use 14 alternative telecommunications services, either wireless 15 or PSTN, to connect the branch to the data centre. 16 Q. You can't tell us when that was? Are we talking months, 17 years after rollout? 18 A. I would have to check when it was deployed, but it -- is 19 this national -- I'm struggling to understand -- 20 Q. National rollout was sort of through 1999 and 2000. 21 2000 was when it really began in a big way. 22 A. Okay. So I would have to check when this manual 23 solution I explained was deployed. I just don't know 24 when it was deployed, but it -- 25 Q. No, all right. Well, is it possible that non-polling 79 1 would have continued as an issue until Horizon Online or 2 is that wrong? 3 A. So the original reason for using ISDN as a network 4 technology, one of the justifications was that most of 5 the transactions didn't require an online connection to 6 be carried out, albeit they did need to synchronise. 7 When the change was made to not do the benefit 8 transactions but to move to Network Banking, then the 9 whole network approach changed and, at that point, we 10 were looking at having backup technology integrated, so 11 there will be a primary network type and a backup 12 network in each branch. 13 Q. So we're talking, are we, about the Post Office's 14 attempts to move into different areas because the 15 Benefits Agency revenue stream was no longer -- 16 A. Yes. 17 Q. -- going to be there? 18 A. And the consequence on the network being that the 19 network had to be there for those transactions to take 20 place, as opposed to it was more a batch system where 21 the transactions could take place and then get 22 synchronised later. So, yes. 23 Q. So Network Banking was going to require being constantly 24 on, was it, as opposed to the intermittent design? 25 A. Yes. 80 1 Q. Did that ever come to pass before Horizon Online? 2 A. Yes, it did. 3 Q. Can you give us an idea of when that was? 4 A. So I've got the timescales here, I can just look them 5 up. 6 (Pause) 7 So the network changes which introduced -- the 8 diagram I'm looking at here starts in 2006, so I'm 9 just -- I don't have the information about exactly what 10 happened before that but, certainly, in 2006 is when we 11 started rolling out the branch network device which had 12 integrated backup. 13 Q. So that was going to be fully on all the time, instead 14 of the polling issue? 15 A. Yes, most definitely, and, in fact, we did introduce 16 fully on much earlier than that. As soon as we went to 17 online banking, we moved away from ISDN intermittent to 18 ISDN nailed up. 19 Q. Again, can you say when that was? 20 A. Not accurately, not without checking, but it would have 21 been prior to introduction of any online banking because 22 it wouldn't have been possible to do it over the ISDN 23 network on demand. 24 Q. All right. So for a period of, presumably, some years, 25 at least, after rollout in 2000, there was still this 81 1 intermittent service with the occasional non-polling 2 incidents; is that right? 3 A. Yes. 4 Q. All right. Well, let me just then -- just a few 5 questions to bottom out what non-polling meant and how 6 it would have affected subpostmasters. 7 So if we look at page 87 of the document that's up 8 and we scroll down, thank you, to a summary of -- that 9 one that's actually at the bottom of the page, so we can 10 stay there. 11 This is a list of extracts from PinICLs and the one 12 that's dated 4 January 2000 explains a sort of a typical 13 example: 14 "'This office is still not polling and hasn't polled 15 for 11 days -- please resolve ASAP.' 'Missing objects 16 relating to EPOSSRec were inserted today by P Carroll. 17 The PO should disappear from the non-polling report 18 tomorrow.'" 19 So what we're seeing there is the effect of 20 non-polling is that one can have missing objects, in 21 other words missing transactions; is that right? 22 A. The ... 23 (Pause) 24 Based on the non-polling report, showing that this 25 particular post office wasn't able to communicate with 82 1 the data centre, then any objects created in the data 2 centre would not have made it to the post office and 3 similarly in the other direction, so that's -- that's 4 what would happen if there was no communication. 5 Q. So the result here is that objects have had to be 6 inserted, in other words transactions have had to be put 7 into the accounts, haven't they? 8 A. Well, I think -- so my immediate thought on reading this 9 is that I recall that, after a number of days of 10 non-polling, there was meant to be a process in place to 11 try and synchronise the post office with the data 12 centre, so that's what would have I expected to be at 13 the normal as-designed solution behaviour for this. 14 In terms of what's happened here, clearly that 15 didn't take place, or wasn't successful, so I can see 16 that the individual -- I know who he is -- is stating 17 that he had to put in some missing data. What I cannot 18 tell from this is whether that missing data was 19 something he had to insert in the data centre, but I -- 20 on the basis that the branch is non-polling, it would 21 mean that it would have to be there because he can't 22 communicate with the branch. 23 But what I cannot tell from reading this is whether 24 this is an approved workaround or whether this is 25 a one-off because the as-designed solution would be for 83 1 someone to attend that office with the special laptop to 2 attempt for it to synchronise with the data centre. 3 Q. Was there a process for making sure this person with the 4 special laptop arrived? 5 A. Yes. There was a whole solution around this. I think 6 it was called Day D solution. 7 Q. Say that again? 8 A. I think -- I recall it was called the Day D solution. 9 Q. Day D solution? 10 A. Yes. 11 Q. Would the subpostmaster in the run-up to this have 12 received any alert or any message? 13 A. I'm not sure what the operational service was around how 14 this was deployed. I mean, clearly, to gain access to 15 the post office, there would have to be some kind of 16 communication, but I'm not sure what the service process 17 was. 18 Q. Absent a human intervention, somebody arriving with the 19 special laptop, was there any system built in, 20 automated, if you like, that would tell postmasters when 21 they weren't polling? 22 A. I don't know. There certainly could have been, very 23 easily, but I don't know if that was actually deployed 24 on the counter because, clearly, the -- any -- it would 25 be very easy on the counter to detect that this is 84 1 happening, but whether it was put in place or not, 2 I don't know. 3 Q. Who would have been responsible for that? 4 A. So that would be as part of the counter development 5 team. So the -- so I think that would be -- at the 6 time, Gareth was the counter and Riposte TDA so he would 7 have been aware of that, or it could have been one of 8 the application people. I'm really not sure even if 9 there was anything put in place like that. 10 Q. Was there any liaison with your team over thinking 11 through the implications of this, so that -- your team 12 obviously being responsible for the network side of it 13 and Gareth's team thinking about it from the point of 14 view of the counter application, was there effective 15 liaison to make sure that subpostmasters would receive 16 the right sort of messages that might say, for example, 17 "You haven't polled for a number of days, there's a risk 18 of missing transactions"? 19 A. I'm not sure if that took place or not. 20 Q. You don't recall, anyway, having those kind of 21 conversations? 22 A. No, and I probably wouldn't have been involved in them 23 if there were, so I wouldn't expect to be involved in 24 them. 25 Q. Who would have been involved with them on the network 85 1 side? 2 A. So we had network designers. At the time of doing that 3 solution, that was David Tanner, so from a network 4 design point of view, it would have been him. It would 5 also have been customer service because, this is 6 a service-related matter, so they would have been 7 involved. 8 MS PAGE: Thank you. 9 MR BLAKE: Thank you, sir. I don't think there are any 10 other questions. 11 SIR WYN WILLIAMS: All right. 12 Well, thank you, Mr Jarosz, for coming to the 13 Inquiry and answering all the questions which were put 14 to you. I'm grateful. 15 A. Thank you. 16 MR BLAKE: Thank you, sir. Are you content for us to all 17 take an hour's lunch now rather than starting the next 18 witness and interrupting? 19 SIR WYN WILLIAMS: Of course, yes. 20 MR BLAKE: Thank you, sir. Perhaps we could come back at 21 1.30. 22 SIR WYN WILLIAMS: Yes, by all means. 23 MR BLAKE: Thank you very much. 24 (12.32 pm) 25 (The luncheon adjournment) 86 1 (1.30 pm) 2 MS HODGE: Good afternoon, sir. 3 SIR WYN WILLIAMS: Good afternoon. 4 MS HODGE: We're just waiting for the next witness to 5 attend. 6 SIR WYN WILLIAMS: Yes, that's fine. 7 (Pause) 8 MS HODGE: Sir, our next witness is Mr Jeram. Please could 9 the witness be sworn. 10 MR PETER JERAM (sworn) 11 Questioned by MS HODGE 12 MS HODGE: Please give your full name. 13 A. Sorry? 14 Q. Please give your full name? 15 A. Peter Ernest Jeram. 16 Q. Mr Jeram, you should have in front of you a witness 17 statement dated 6 August of this year -- 18 A. Yes. 19 Q. -- is that right? The statement runs to nine pages. 20 Could I ask you please to turn to page 8 of that 21 statement. 22 A. Yes. 23 Q. Do you see your signature there -- 24 A. I do, yes. 25 Q. -- at the bottom of the statement. Is the content of 87 1 the statement true to the best of your knowledge and 2 belief? 3 A. Yes. I've got one comment on it, if that's okay? In 4 section 15, when I read that again, when it says in 5 there about the cash account, and I made a statement 6 saying "and cannot therefore comment on whether there 7 were issues", I was talking about issues that we didn't 8 know about in my role and support in the end to end and 9 MOT. I did know about issues that were found and then 10 resolved. I just wasn't sure that was clear on that 11 statement. 12 Q. So what you're saying your evidence is, that at 13 paragraph 15 of your statement, you were saying that you 14 were not aware of issues of which you were not aware; is 15 that, in effect, your correction? 16 A. I guess so, yes. This implied I didn't know about 17 anything and we did have issues and we did correct 18 issues. So ... yes. 19 Q. Thank you. I'm going to begin by asking you some brief 20 questions about your recruitment by ICL Pathway. You 21 joined ICL Pathway as a release project manager in 22 approximately 1997; is that correct? 23 A. Yes. 24 Q. At that stage, you were not an employee of 25 ICL Pathway -- 88 1 A. Correct. 2 Q. -- but you had been recruited to join the programme via 3 a IT consultancy; is that right? 4 A. Yes. 5 Q. You later became a permanent employee of what became 6 known as Fujitsu Services Limited in or around 7 April 2003; is that right? 8 A. Yes. 9 Q. You remain employed by Fujitsu today; is that correct? 10 A. Yes. 11 Q. But not on projects related to Horizon, I understand? 12 A. No. 13 Q. It is in your capacity as a current employee of Fujitsu, 14 who had direct involvement in the matters to which this 15 Inquiry relates, that you were invited to provide 16 a witness statement to the Inquiry on behalf of Fujitsu; 17 is that right? 18 A. Yes. 19 Q. The purpose of that statement was to assist the Inquiry 20 with the matters canvassed in two Rule 9 requests, the 21 first dated 11 March of this year; is that correct? 22 A. Yes. 23 Q. And the second, 1 July? 24 A. Okay. 25 Q. Those requests covered a range of issues, which included 89 1 issues identified in the development of the cash account 2 function Horizon, and you have referred just now to 3 paragraph 15 of your statement -- 4 A. Yes. 5 Q. -- which was directed at that, as well as the accuracy 6 and integrity of the data recorded and processed on the 7 Horizon System, and the extent to which deficiencies 8 with Horizon were capable of causing apparent 9 discrepancies or shortfalls in branch accounts. 10 Those were the three areas canvassed in those 11 requests, were they not? 12 A. Sorry, I don't remember the detail of the request. 13 I was certainly asked some questions which I answered. 14 Q. Bearing in mind what you have just told us about 15 paragraph 15 of your statement, do you consider you have 16 been candid in your statement to the Inquiry about your 17 knowledge of technical issues with Horizon at the time? 18 A. I would say I remember more now from the bundles I have 19 been provided than maybe I did at the time of the 20 statement. 21 Q. Before you finalised your statement, you were invited to 22 refresh your memory from the contemporaneous records 23 held by Fujitsu, were you not? 24 A. I was certainly given some documents to remind myself on 25 things, yes. 90 1 Q. You had access to all of the documents in Fujitsu's 2 possession, did you not, that were relevant to your 3 involvement? 4 A. I don't know. I was certainly -- had access to some 5 documents that were provided to me, yes. 6 Q. Did you ask to be provided with all documents that were 7 relevant to your involvement in the period prior to the 8 rollout of Horizon? 9 A. When I had some questions on things I asked and was 10 provided with a document, yes. 11 Q. I wonder if you could help us then. How is it that you 12 came not to mention the issues that you say you have now 13 come to understand in the recent disclosure that's been 14 provided to you? 15 A. It's more a case of reading the wording that I put in 16 there because, for example, I got involved in the end to 17 end testing and the model office rehearsals and testing 18 with Post Office and, through that, there would have 19 been incidents that were raised on the cash account and 20 incidents that were cleared, so I would have had the 21 visibility of those taking place at that time. 22 Q. You specifically mention in your statement, at 23 paragraph 26, that you were aware of a number of formal 24 internal audits of the Horizon System; is that correct? 25 A. Yes. 91 1 Q. Did you ask to see copies of those audit reports before 2 finalising your statement? 3 A. Yes, I did see some. 4 Q. We will return to some of those a little later. In your 5 role as release project manager, I understand you were 6 responsible for project managing the release of software 7 by ICL Pathway into the live estate; is that correct? 8 A. Yes. 9 Q. You have explained in your statement that this was not 10 a technical role, as far as you were concerned -- 11 A. Yes. 12 Q. -- and that you relied on those who did have the 13 relevant technical expertise to bring technical matters 14 to your attention; is that right? 15 A. Yes. 16 Q. You have also stated you were not directly involved on 17 the technical side of the development of the project; is 18 that right? 19 A. Yes. 20 Q. You were, however, notified of significant technical 21 developments and issues which affected the timing and 22 release of software; is that correct? 23 A. Yes. 24 Q. Presumably, knowledge of such technical issues would 25 have been critical for you to perform your role as 92 1 a project manager? 2 A. To a certain level, yes. 3 Q. Though not a technical expert, you presumably had quite 4 a high level understanding of the purpose and function 5 of the key components of Horizon; is that correct? 6 A. Probably at that time, yes. 7 Q. You would have known, therefore -- but please correct me 8 if I'm assuming too much -- but I presume you would have 9 known that the Electronic Point of Sale Service, one of 10 the key components of the Horizon System, was 11 responsible for recording and processing all of the 12 transactions carried out within the Post Office branch 13 by customers purchasing goods and products of the Post 14 Office; is that right? 15 A. Yes. 16 Q. You would have known it was responsible for balancing 17 receipts and payments -- 18 A. Yes. 19 Q. -- and for producing what was known as the cash account; 20 is that right? 21 A. Yes. 22 Q. Presumably, you knew that the essential function of the 23 cash account was to produce the definitive weekly 24 summary of all the transactions recorded within the post 25 office branch -- 93 1 A. Yes. 2 Q. -- and that the cash account function, therefore, served 3 an essential accounting function, both for the Post 4 Office and for its agents who were using the system? 5 A. Yes. 6 Q. I would like to ask you some questions about a report 7 which was produced in September 1998 on completion of 8 what was known as the EPOSS PinICL task force. The 9 report to which I'm referring bears the unique reference 10 number FUJ00080690. You have had an opportunity to read 11 this report, have you? 12 A. Yes. 13 Q. Do you recall being shown a copy of this report at or 14 around the time it was produced in September 1998? 15 A. No. 16 Q. Were you aware in the summer of 1998 that the volume of 17 PinICLs recorded against the Horizon product was very 18 high? 19 A. The timing not sure but, yes, I know there were lots of 20 PinICLs at some stage. 21 Q. Did you know at the time that the PinICL count was 22 sufficiently high that the task force had been 23 established in an effort to reduce it? 24 A. Yes, I think I do. 25 Q. Were you made aware, on completion of the task force, 94 1 about the concerns which had been expressed by those 2 with the relevant technical expertise about the quality 3 of the EPOSS code? 4 A. There was certainly a discussion about -- yes, the 5 quality of the product. 6 Q. Was that a discussion that you had in one of the 7 development directors meetings, which you attended with 8 Terry Austin? 9 A. Probably. 10 Q. What exactly did you understand to be the nature of the 11 concerns about the quality of the EPOSS code? 12 A. With the large number of PinICLs that had been raised 13 through the testing services. 14 Q. That indicated what, as you understood it? 15 A. That the product was of questionable quality. 16 Q. Were you aware that those with relevant technical 17 expertise had expressed fears that the application of 18 PinICL fixes was likely to lead to yet further 19 degradation of the quality of the EPOSS code? 20 A. Where -- we were aware at the time that would be a risk 21 with the number of changes that were being made. 22 Q. So that was something of which you would have been aware 23 at the time? 24 A. Yes. 25 Q. You mentioned, when confirming the content of your 95 1 statement, that you had some oversight of model office 2 and end to end testing; is that right? 3 A. Yes. 4 Q. We know that you participated in a review in early 5 December 1998 of the PinICLs raised during what was 6 known as MOR3 and end to end testing, which had been 7 carried out in November of that year; that's right, 8 isn't it? 9 A. Yes, to my knowledge, yes. 10 Q. You have been shown a copy of the memorandum that was 11 produced by Andrew Simpkins on 4 December 1998 in 12 connection with that review, have you not? 13 A. I have looked at all the things sent to me, so if that 14 was one of them, yes. 15 Q. The document to which I'm referring is POL00028429, 16 please. We can see that you are named as one of 17 a number of recipients of that memorandum, are you not? 18 A. Yes. 19 Q. Have you taken the opportunity to refresh your memory of 20 the document? 21 A. This particular one versus the others, I can't remember 22 exactly what this one said. 23 Q. The memorandum canvasses a number of issues that were 24 identified during the review. If we could scroll down, 25 please, to the first page, a little bit lower on the 96 1 first page. Under the heading "Progress this Week", 2 Mr Simpkins confirms: 3 "As you are aware Horizon, TIP [which would be 4 transaction information processing] and Pathway have 5 carried out a comprehensive and detailed analysis this 6 week of PinICLs arising from MOR3 and E2E ..." 7 End to end. So those two testing cycles; is that 8 right? 9 A. Yes. 10 Q. "... (and outstanding faults from previous phases). We 11 would like to thank Pete Jeram for his active support to 12 this review. I attach a copy of the summary totals and 13 of the full PinICL analysis pack." 14 If we go on a page please to the second page, under 15 the heading "Testing Issues", it reads: 16 "Since tabling the paper on the 'Key Problem Area 17 Analysis' at the Checkpoint meeting on 18th November 18 good progress has been made on most of the 9 areas 19 identified. We will reissue this summary next week 20 showing the current action points. Specific concerns 21 that have been confirmed by the PinICL review 22 include ..." 23 Then we can see a list of issues. Firstly, those on 24 the transaction information processing interface, these 25 include "Inconsistencies between the transaction file 97 1 totals and the cash account", that's the third bullet 2 point. Can you see that? 3 A. Yes. I wouldn't have been involved in the detail of 4 these things. The action I was doing was making sure 5 there was a review and that everybody was being very 6 open and sharing them. 7 Q. Well, we can see you participated in that review -- 8 A. Yes. 9 Q. -- and Mr Simpkins expressed his thanks to you for 10 assisting. 11 A. But it wouldn't have been my thanks from diagnosing the 12 issues, or the things. 13 Q. No, forgive me, I'm not suggesting that you would have 14 had a detailed technical awareness of the underlying 15 causes of these issues, but you would have been aware, 16 surely, by virtue of that review and the receipt of this 17 memo, that issues of this nature were being discussed at 18 the time -- 19 A. Yes. 20 Q. -- in December 1998? 21 A. Yes. 22 Q. So we have there, at the third bullet point: 23 "Inconsistencies between the transaction file totals 24 and cash account ..." 25 At the fourth bullet point: 98 1 "'Lost' BES [that would be Benefit Encashment 2 Service] transactions on the transaction file." 3 A little further down, there is reference under the 4 heading "On the Counter" to a number of incidents around 5 stock unit balancing. Then, if we scroll down a little 6 further, please, under the heading "Other Issues", we 7 see a number listed, the first of which are "cash 8 account balances", and there's reference there to 9 a constructive joint meeting on the reasons for 10 imbalances and the action being taken to address these. 11 A. Yes. 12 Q. I think we're agreed that, in December 1998, these were 13 all issues that were certainly on your radar? 14 A. Yes. 15 Q. You also knew, did you not, that there remained quite 16 significant concerns about Horizon's accounting 17 integrity at the point at which the system was accepted 18 by the Post Office in late September 1999. 19 A. Although I wasn't directly involved in the acceptance, 20 I know there were issues that were going through 21 discussion, exactly. 22 Q. The concerns about Horizon's lack of accounting 23 integrity were sufficiently serious at that stage, were 24 they not, that ICL Pathway had agreed to produce a new 25 piece of software to perform reconciliation checks? You 99 1 were aware of that, were you not? 2 A. Yes, that was towards the end of 2000 maybe. No -- yes, 3 by the end of 1999, yes, yes. 4 Q. The purpose of that software, known as the accounting 5 integrity control release, was to detect cash account 6 imbalances and to produce reports to enable them to be 7 rectified -- 8 A. Yes. 9 Q. -- is that not correct? 10 A. Yes. 11 Q. You were, in fact, responsible for project managing the 12 release of that piece of software, were you not? 13 A. Yes. 14 Q. We can see that if we pull up document FUJ00118156 15 please. Forgive me, it's 156 please. 16 (Pause) 17 Thank you. Sorry, that reference error was my 18 fault. 19 This document is described as a process release 20 note. 21 A. Mm-hm. 22 Q. It is dated 29 October 1999. We can see at the top it 23 is version 0.1. It: 24 "Provides a definition of the CSR+ Increment 2.2 25 [relating] to Acceptance Incident 376 Release for [Post 100 1 Office Counters Limited]." 2 A. Yes. 3 Q. A document that was reviewed by you -- 4 A. Mm-hm. 5 Q. -- and to which you contributed at the time? 6 A. Yes. 7 Q. Before we move on, please, to another topic, I would be 8 grateful if you could assist me with one further 9 document. This is document FUJ00118175, please. This 10 is a document which was produced to the Inquiry by 11 ICL Pathway. You have been shown a copy of this 12 document, I believe. 13 A. I have, yes. 14 Q. Have you taken the opportunity to read it? 15 A. I have. I wouldn't say I fully understand it but ... 16 Q. It's clear from its title that it relates to EPOSS 17 reconciliation issues and Acceptance Incident 18 number 376. 19 A. Yes. 20 Q. There's an entry at the top which indicates that 21 comments have been added to the document by "P JP" can 22 you help us with who is that a reference to you? 23 A. I don't think -- when I read it and I saw that I thought 24 it might be John Pope. 25 Q. Thank you. I would like now, if I may please, to turn 101 1 to the development audit of the Core System Release 2 Plus, which was conducted in September 1999. This was 3 an audit carried out by Jan Holmes, Pathway audit 4 manager, who produced a report recording his findings in 5 late October 1999. 6 A. Yes. 7 Q. You were made aware at the time of the findings of that 8 audit, were you not? 9 A. Yes. 10 Q. Do you recall reading the audit report? 11 A. Yes. 12 Q. You have recently been provided with a copy. Have you 13 taken the opportunity to refresh your memory -- 14 A. Yes. 15 Q. -- of its contents. It bears the reference FUJ00079782, 16 please. Have you read, in particular, the section of 17 the report at pages 19 and 20 which addressed the 18 author's findings in relation to EPOSS? 19 A. I read the report, so I would have gone through that as 20 well, yes. 21 Q. Whether or not you were shown a copy of the report into 22 the EPOSS PinICL task force in September 1998, you would 23 have known, upon reading this audit report, that the 24 EPOSS PinICL task force report had the previous year 25 called into question the maintainability and resilience 102 1 of the EPOSS code -- 2 A. Yes. 3 Q. -- and that was by reason of the high number of PinICL 4 fixes which had been applied to the EPOSS product -- 5 A. Yes. 6 Q. -- that's correct, isn't it? What's more, you would 7 have also known, on reading this report, that, since 8 completion of the task force, nearly 1,000 PinICLs had 9 been raised against the EPOSS product and that the 10 application of fixes to address those faults was bound 11 to have worsened the quality of the code? 12 A. There was that risk, yes. 13 Q. Jan Holmes' concerns about the quality of the EPOSS 14 product were sufficiently grave that he recommended that 15 consideration be given to redesigning or rewriting 16 EPOSS? 17 A. Yes. 18 Q. You were aware of that? 19 A. Yes, I saw that in the report, yes. 20 Q. This wasn't the first occasion on which that 21 recommendation had been made, had it? 22 A. I don't remember. 23 Q. We know from reading this audit report that an earlier 24 report addressing the quality of the EPOSS product had 25 been produced by Pathway on 21 September 1999. Were you 103 1 aware of that report? 2 A. I don't know. 3 Q. Perhaps if we could turn to page 2, please. If we 4 scroll down, please, thank you. Under the heading "0.3 5 Associated documents", there's a reference at point 7 to 6 a report on EPOSS solutions, dated 21 September 1999. 7 Were you aware of that report? 8 A. I don't remember that report. 9 Q. We see that report referenced in this audit. If we 10 could go to page 20, please. In the box there, the 11 audit states that: 12 "The EPOSS Solutions Report [document number 7 in 13 the associated documents we saw just a moment ago] made 14 specific recommendations to consider the redesign and 15 rewrite of EPOSS, in part or in whole, to address the 16 then known shortcomings." 17 So that recommendation was first made on 18 21 September 1999. Do you know whether or not a copy of 19 that report was provided to Post Office Counters prior 20 to their decision to accept the Horizon decision in late 21 September? 22 A. I don't know. 23 Q. Do you consider that a copy of that report should have 24 been provided to Post Office Counters to inform their 25 decision about acceptance of the Horizon System? 104 1 A. I think so, along with the testing from it, yes. 2 Q. Who do you think was responsible for ensuring that was 3 done within ICL Pathway? 4 A. It probably would have been done through one of the 5 reviews that Mike, Terry and I were at. 6 Q. Forgive me, but by "reviews" do you mean internal 7 reviews -- 8 A. No, the meetings we had with Post Office. 9 Q. With Post Office? 10 A. Yes. I don't think we did, but that, I guess, would be 11 the place that it was shared. 12 Q. That's where it ought to have been shared, is your view? 13 A. Yes. 14 Q. When you received a copy of this CSR+ development report 15 in late October 1999, did you take any steps to bring 16 its findings and recommendations to the attention of 17 Post Office Counters? 18 A. I don't remember doing so. 19 Q. Do you consider that a copy of the CSR development audit 20 report should have been provided to Post Office Counters 21 to inform their decision about the resolution of 22 Acceptance Incident number 376? 23 A. I think we concentrated more on the testing that showed 24 that it was working, necessarily, than the report, but 25 as we had quite an open relationship then -- yes, 105 1 I don't know why we wouldn't have shared it. 2 Q. Is it your evidence that you think you would have shared 3 it at the time? 4 A. No, my evidence is I don't know. 5 Q. You just don't know. The CSR+ development audit report 6 was supported by a schedule of corrective actions in 7 which the recommendations resulting from the audit were 8 recorded and agreed corrective actions were documented. 9 Were you aware of that? 10 A. Yes. 11 Q. One of those recommendations we can see here was that, 12 in light of the continued evidence of poor product 13 quality", that is to say in the EPOSS product, that the 14 recommendations to consider the redesign and rewrite of 15 EPOSS be reconsidered. 16 So you were aware, were you not, that Jan Holmes had 17 specifically recommended that that earlier 18 recommendation be reconsidered? 19 A. From reading this, yes, I'm sure, at the time. 20 Q. You have received a copy of the schedule of corrective 21 actions that was circulated in late November -- forgive 22 me, at the time you would have received a copy of the 23 schedule of corrective actions? 24 A. Yes. 25 Q. That's right. For the benefit of the transcript, that 106 1 document bears the reference FUJ00079783. 2 You also received a copy of the revised schedule in 3 May 2000; is that right? 4 A. Yes. 5 Q. Have you refreshed your memory of those documents -- 6 A. Yes. 7 Q. -- from the copies provided to you? 8 A. Yes. 9 Q. Please could WITN04600104 be shown on the screen, 10 please. So this is version 2.0, so the version dated 11 10 May 2000 and we can see you there named on the 12 distribution list. 13 If we could please scroll down to page 10 of the 14 schedule -- forgive me, page 9 please. Under the 15 heading "Report Observation/Recommendation" we can see 16 reference to the recommendation to reconsider the 17 redesign and rewrite of EPOSS, that's right? 18 A. Yes. 19 Q. You are not the owner of that action and you are not 20 named as one of the management team members. That's 21 right, isn't it? 22 A. Correct. 23 Q. You were, however, involved in its resolution, were you 24 not? 25 A. Yes. 107 1 Q. We know from the entries in the agreed actions column, 2 so the second from the right, that Terry Austin had, on 3 15 November, requested that the recommendation to 4 redesign and rewrite the EPOSS application be closed, 5 having concluded that it would be difficult to justify 6 the case for rewriting it. You were presumably aware of 7 that, were you? 8 A. Yes. 9 Q. At the bottom of the page we can see that Mr Austin 10 proposed continuing to monitor the PinICL stack for the 11 next few months to assess whether or not it was 12 necessary to re-evaluate that decision. 13 A. Yes. 14 Q. Were you aware of that at the time? 15 A. Yes. 16 Q. So, essentially, subject to what the PinICL stack 17 showed, the recommendation was either going to be closed 18 or indeed taken further? 19 A. Yes. 20 Q. If we scroll on, please, to page 10, we can see two 21 entries dated 8 December 1999. One of these appears to 22 relate to you, that's the second of those entries. The 23 first reads: 24 "JH requested statistics on fixes delivered to live 25 from RM. Also informed TPA that requires agreement of 108 1 MJBC before this can be closed." 2 We understand the reference to JH to be Jan Holmes, 3 the author of the report. 4 A. Yes. 5 Q. Does that sound correct? 6 A. Yes. 7 Q. He had requested statistics on fixes. This is 8 presumably a reference to statistics from release 9 management; is that right, "RM"? 10 A. I was wondering if it meant that, but it could do, yes. 11 Q. That would be statistics related to software fixes 12 delivered to the live system; is that right? 13 A. Correct, or into the testing phases ready to go to the 14 live system. 15 Q. We can see there that Jan Holmes has informed Terry 16 Austin that his instruction to close the recommendation 17 in fact requires the agreement of the programme 18 director, Mike Coombs. 19 A. Yes. 20 Q. That was correct, isn't it? The second entry, dated 21 8 December reads "MJBC", which we know is Mike Coombs: 22 "... Confirmed that unless RM statistics 23 contradicted reports provided by PJ the recommendation 24 could be closed." 25 So we know, on the one hand, there's this request 109 1 for statistics from release management, but there's also 2 a reference here to reports provided by you. Can you 3 help us, please? 4 A. Yes, it would be the same data, so I would have given 5 input on PinICLs that had been closed or been addressed 6 and this is asking for confirmation that release 7 management agree with the data that I had. 8 Q. Can I just clarify, you were, of course, the release 9 project manager, why might it be that there were 10 contradictions between your reports and the statistics 11 held by release management? 12 A. No, there wouldn't be. 13 Q. Right. 14 A. He has asked for -- I think, in that, Mike is asking 15 for -- formalising release management providing the data 16 that matches. 17 Q. Okay, and that's to satisfy him -- 18 A. Yes. 19 Q. -- that it is, indeed, proper to close that action? 20 A. Yes. 21 Q. There appears then to be a gap of approximately 22 four months and we see the next entry is dated 23 7 April 2000; can you see that? 24 A. Yes. 25 Q. In the meantime, concerns were raised with you in early 110 1 January 2000 about an increasing number of PinICLs, cash 2 account misbalances and reconciliation errors, were they 3 not? 4 A. I don't remember. I'm not saying it didn't happen, but 5 I don't remember. 6 Q. I'm referring to emails dated early January 2000, which 7 bear the reference FUJ00079332. Please could that be 8 shown on the screen. 9 The author of this email is a Duncan MacDonald. Was 10 he one of the technical experts on whom you relied to 11 bring technical issues to your attention? 12 A. I don't remember Duncan. 13 Q. The email is addressed to you. We can see at the top 14 there it is dated 4 January 2000, I believe -- 15 A. Yes. 16 Q. -- although quite possibly it could be 1 April. I'm 17 conscious that some emails have the month first and the 18 day second but, be that as it may, whether January or 19 April, we can see here the subject matter of the report 20 is "CI4 Transaction Mode Problems". 21 A. Yes. 22 Q. Do you see that? 23 A. Yes. 24 Q. I understand CI4 was the name of a software release 25 relating to the EPOSS application which was later 111 1 introduced -- 2 A. I don't know if it was just EPOSS, but I think it was 3 an increment forward to the core release. I think it 4 was something like that. 5 Q. So it related to functionality affecting EPOSS and other 6 components of the system? 7 A. Potentially, yes. 8 Q. I think it follows that CI4 had not been released into 9 the live estate at this stage; is that correct? 10 A. I would read that -- 11 Q. Taking this to be January or indeed April? 12 A. Because it is talking about end to end systems, so 13 I think it's having problems in its testing. 14 Q. We're in the testing phase? 15 A. Yes. 16 Q. CI4 was part of the larger release known as the Core 17 System Release Plus; is that correct, to your 18 recollection? 19 A. I don't know whether CI4 is part of that or whether CI4 20 was an increment to the release before that, which -- 21 Q. The CSR? 22 A. Yes, because I think CSR+ didn't happen until quite 23 a bit later. 24 Q. I think that was rolled out in the course of 2000, CSR+. 25 A. Was it? Okay. 112 1 Q. I believe. We can check. Be that as it may, we're 2 dealing here with issues identified in software testing 3 and this email reads: 4 "We are getting an increasing number of PinICLs on 5 the end-to-end system handling of the new CI4 6 transaction modes ..." 7 These are described in brackets as "PT", "NAD", 8 "RIAD" and "ROAD": 9 "... leading to cash account misbalances and 10 reconciliation errors. These PinICLs are generally 11 being batted about between the different areas. 12 "I suggest a workshop is set up, led by either 13 Requirements or EPOSS, to present the current end-to-end 14 solution, identify the problem areas and then agree the 15 necessary changes to achieve a consistent solution. 16 This may involve having to get clarification of 17 requirements from POCL. 18 "If anyone can think of a better approach or that 19 there isn't a problem please say so." 20 Do you recall whether the proposal to set up 21 a workshop, whether or not that proposal was taken up? 22 A. I would have thought so. It's quite an obvious thing to 23 be suggesting and there obviously were problems that 24 needed to get together and work out what to do. 25 Q. Given that this software release CI4 related, at least 113 1 in part, to the EPOSS application and there were known 2 to be PinICLs causing cash account misbalances and 3 reconciliation errors, did this email cause you to 4 consider whether or not the outstanding recommendation 5 to redesign and rewrite the EPOSS application ought to 6 be re-evaluated? 7 A. I think following the workshop and seeing what's 8 happening in the testing, what's being found, would have 9 led into that decision about how bad it was. 10 Q. Did this increasing number of PinICLs and the cash 11 account misbalances it was causing not, in itself, call 12 into question the earlier decision of Terry Austin to 13 close the action? 14 A. Depending on what was found through reviewing this then 15 that might have ultimately led to that. 16 Q. Pathway's concerns about the quality and stability of 17 the Horizon software were issues of which you continued 18 to have oversight in the spring of 2000, were they not? 19 A. I think it was about this time that I changed my role. 20 I was asked to look after the development teams. There 21 were some other challenges and we put a corrective 22 action plan in place on certain areas. I'm not sure 23 whether I continued at that point to still be the sort 24 of release project liaison or not. 25 Q. We can see you did have some oversight of these issues. 114 1 You have been provided with a copy of ICL Pathway's 2 consolidated risk register -- 3 A. Yes. 4 Q. -- covering the period of approximately May 1998 to 5 May 2000; is that correct? 6 A. Mm-hm. 7 Q. For the benefit of the transcript that document bears 8 the reference FUJ00077884. Apologies, we're just 9 verifying a reference. 10 (Pause) 11 Sir, I wonder if you would mind if we take a short 12 five-minute break to see if we can enable the document 13 to be shown on the screen? 14 SIR WYN WILLIAMS: Well, of course. Just before we do that, 15 Mr Blake and I have been in email communication about 16 tomorrow and am I right in thinking now that the 17 witnesses scheduled for tomorrow are either not going to 18 give oral evidence or be called at some future time? 19 MS HODGE: Sir, that's correct. Mr Jeram is our last 20 witness for this week and I certainly would hope -- 21 SIR WYN WILLIAMS: All right, fine. I just wanted to say 22 that publicly, as quickly as possible, so that anybody 23 listening would know that. So at the end of this 24 afternoon's session we won't be convening tomorrow, we 25 will be convening next Tuesday? 115 1 MS HODGE: Thank you, sir, yes. 2 SIR WYN WILLIAMS: Fine, thank you. Let me know when you're 3 ready. 4 MS HODGE: Thank you. 5 (2.13 pm) 6 (Short Break) 7 (2.23 pm) 8 MS HODGE: Sir, thank you for the additional time. We have 9 managed to display the document to which I was referring 10 a short time ago. This is a copy of the consolidated 11 risk register produced by ICL Pathway in the period 12 May 1998 to May 2000. What you can see, I hope, on the 13 screen is page 4 of that risk register where two entries 14 are recorded. 15 The first bears the reference 00_25. That was 16 a risk raised in February 2000 of which Terry Austin was 17 the owner. 18 Under the heading "Risk Summary", it is described as 19 a maintainability -- forgive me, that's the second of 20 the two entries. 21 The first is 00_38, also raised in February 2000 and 22 of which Terry Austin was also the risk owner. It's 23 a risk in the area of development, which bears the title 24 "Maintenance activity". For the benefit of the witness, 25 if we could scroll, please, to the right-hand side, 116 1 there are some further columns. At column N, Mr Jeram, 2 we see that you were the mitigation owner of that risk; 3 is that correct? 4 A. Yes. 5 Q. Thank you. Please could we scroll back so we can see 6 column G, which contains the "Risk Description" and we 7 can see that provides that: 8 "Maintenance effort over the life of the contract 9 exceeds the planned levels. Analysis of call 10 information and user problems necessitates research 11 effort, diverting resources from development or PinICL 12 support work; hence PinICL stacks remain high. Still 13 developing on aged platforms. Skilled and experienced 14 staff increasingly being lost through attrition; no 15 longer able to retain with prospect of developing new 16 applications." 17 Then under the column H, we can see a description of 18 the "Risk Impact", and that provides: 19 "Cannot retain experienced staff. 20 "Cannot attract quality people -- availability. 21 "Increased personnel costs in development as staff 22 are retained for maintenance. 23 "Increased product support costs." 24 The following three columns contain an assessment of 25 the probability, the impact and then the factor of those 117 1 risks. The probability of that risk occurring, that's 2 in relation to maintenance activity, is recorded as 3 being 3. Do you recall that, Mr Jeram? 4 A. I can read it, yes. 5 Q. The score of 3, as I understand it, reflected 6 a probability of 30 to 60 per cent of that risk 7 occurring -- 8 A. Yes. 9 Q. -- is that right? 10 A. Yes, I'm looking at the front page, yes, agreed. 11 Q. The impact of the risk, we can see, was scored as 4 and 12 that reflected a "major change for approved costs, 13 quality, timescale of some activity, which would cause 14 serious delay"; is that right? 15 A. Yes. 16 Q. That gives a factor score of 12. We can see that in 17 column K. Under column L we have a description of the 18 "Mitigation Action" for this particular risk. Thank 19 you. That reads: 20 "8th June Investment Strategy Board -- need positive 21 decisions on future opportunities; POCL need to move 22 away from 'move to the right' culture to realise new 23 business opportunities ... 24 "Need to reconsider contingency plan retaining 25 maintenance team for bespoke software with further 118 1 developments untaken by Large Projects." 2 Can you assist us, please, as the mitigation owner, 3 with that entry? 4 A. I think that mitigation is particularly towards the 5 losing of the skilled resources that understood the 6 products. The point that's being raised in the risk 7 description is that, if those skilled people are just -- 8 I'm not saying PinICL fixing wasn't important. If 9 they're just PinICL fixing, then they might want to move 10 on from supporting Pathway. So that was about what were 11 the next opportunities that would keep people interested 12 in doing what they're doing today, because there's 13 something different in the future. 14 Q. The concern being expressed here, as raised in 15 February 2000, was that the PinICL stack remained high. 16 That's what gave rise to this risk of attrition, is it, 17 in relation to staff? 18 A. Yes, I mean, there was obviously CR+ taking place and 19 there were service increments so there will be continual 20 testing and continual new PinICLs coming along. 21 Q. Just for the benefit of the transcript, by "maintenance 22 activity", would I be right to understand that what this 23 document is referring to is the identification and 24 rectification of PinICLs, bugs, errors and defects in 25 the system, as and when -- 119 1 A. Yes, it could be those that were raised at the end of 2 testing cycles that were allowed to be there when we 3 went live, or they could be coming from new changes 4 being made, or they could be coming from the live 5 service. 6 Q. So a number of different domains -- 7 A. Yes. 8 Q. -- creating pressures on your maintenance team? 9 A. Yes. 10 Q. The second entry we see there at row 17, dated 11 February 2000, is described as "Maintainability". It 12 bears the heading "Maintainability" and is described in 13 column G as a risk relating to the quality of software. 14 It states: 15 "Products have grown organically so product 16 stability is not assured." 17 The risk impact reads that there is "Increased 18 costs, operational system failures and reputation". 19 Presumably that's damage to the reputation of 20 ICL Pathway; is that right? 21 A. Yes, and to the reputation of the system. 22 Q. The probability of that risk is assessed as 2, which 23 I understand to mean that it was considered to bear 24 a 10 to 30 per cent likelihood of eventuating; is that 25 right? 120 1 A. Yes. 2 Q. It also carries the impact factor of 4, the same as the 3 maintenance activity risk. The mitigation action is 4 simply to monitor issues in the live estate; is that 5 right? 6 A. That's what it says, yes. 7 Q. Do you recall your ownership of that mitigation at the 8 time? 9 A. Well, it's not really much of a mitigation really, from 10 what I'm reading there. I mean, parallel with this, we 11 had a corrective action plan that was taking place that 12 was a sort of -- quite a big exercise that was in place, 13 going back over designs and software quality and things 14 and sorting out across the estate the supportability for 15 the longer-term of the Horizon product. 16 Q. Forgive me, can you assist us, by a "corrective action 17 plan" are you referring to the plan that we have seen 18 already or are you suggesting that there was another 19 specific plan in place? 20 A. It came about from the audit, from the development 21 audit, that we put a corrective action plan in place 22 because there was discussion about -- at one point of 23 what it was going to cost to do it, which wasn't 24 a problem. 25 Q. Cost to do what, sorry? 121 1 A. To do the work, the extra work. 2 Q. This is to say the maintenance work or the redesign? 3 A. No, to do improving the documentation of the product 4 that was being maintained. 5 Q. Sorry, I don't entirely follow. Improving the 6 documentation of the product? Are we referring here to 7 design documentation or are we -- 8 A. Design and test scripts for retesting, et cetera. 9 Q. And to which product are you referring? 10 A. A number of different products, I think. 11 Q. Right. Forgive me, what we're dealing with here, 12 I think, is issues in relation to the quality of the 13 software. 14 A. Yes. 15 Q. We don't see any reference to a corrective action plan 16 here, do we? 17 A. No. It's only that I know that that would have been 18 happening around the same sort of time, but I'm 19 surprised it's not there as a mitigation. Sorry, that's 20 my point. 21 Q. Right. What emerges, I think, from reviewing this risk 22 register is that there remained significant concerns in 23 the spring of 2000 about the quality of the Horizon 24 software; is that fair? 25 A. Yes. 122 1 Q. And about the ability of ICL Pathway effectively to 2 maintain Horizon in the live estate? 3 A. That was the risks that were being recorded, yes. There 4 was the risk of that, whereas we can see the probability 5 that -- the view was it was a reasonably high 6 probability. 7 Q. I would like to return at this point, please, to the 8 schedule of corrective actions which we reviewed a short 9 time ago. That's the document that bears the reference 10 number WITN04600104, at page 10, please. Thank you. 11 Forgive me, internal -- it says page 11. 12 There are three remaining entries in the right-hand 13 column, the first of these is dated 7 April, by which 14 I assume we are now into the year 2000. That reads: 15 "Email to MJBC [Mike Coombs], TPA [Terry Austin] & 16 PJ [that would be you] ..." 17 A. Yes. 18 Q. "... providing details of RM EPOSS fixes to live." 19 So release management EPOSS fixes to live: 20 "Asked for confirmation that matched PJ reports. If 21 does then will close." 22 So it appears to show that Jan Holmes has obtained 23 details of the RM, the release management, EPOSS fixes 24 to live and is seeking confirmation that these are 25 matching your reports; is that correct? 123 1 A. Yes. I'm surprised it's four months later than the 2 original entry, but if it's the same -- it feels a bit 3 out of date by then. But, yes, that's what it's 4 implying, yes. I assume it's an updated position by the 5 end of -- the beginning of April but ... 6 Q. That is dated 7 April. I think the document to which 7 I referred you a short time ago, which I initially 8 thought was dated January, was in fact dated 1 April. 9 Perhaps if we could just pull that up again to ensure we 10 have an accurate record on the transcript. That is 11 FUJ00079332. 12 If I can just refer you back to the date -- 13 A. Yes. 14 Q. -- this is recorded as sent 4/1/2000. Having looked at 15 some of the other ICL Pathway emails, it appears that 16 they bear first the month, then the day and the year. 17 I don't know if you can assist us with that, whether you 18 think that is correct, that this would have been early 19 April? 20 A. I don't remember the dates being in American format but 21 I'm happy to accept -- I have seen some that obviously 22 are in American format. 23 Q. Thank you. Can we return, please, to the schedule of 24 corrective actions, which is WITN04600104. As you say, 25 some time has elapsed between the last entry on 124 1 8 December and this further entry on 7 April. The next 2 entry, dated almost a month later, is 3 May and this is 3 to record that a reminder email was sent to the above, 4 by which I understand to mean to Mike Coombs, Terry 5 Austin and to you, seeking early response, chased on the 6 same day. Does that assist you at all in your 7 recollection of the progress of this action? 8 A. No, but looking it through, it seems that, for whatever 9 reason, it took Jan longer to get his data to confirm 10 what I had said back in December. I presume the email 11 was asking for Mike to confirm the position and then Jan 12 has had to chase it again. I'm guessing that's what 13 happened. 14 Q. Do you recall what your reports were at this stage, in 15 relation to the volume of EPOSS fixes to the live 16 estate? 17 A. I don't know but it would have been based on -- as 18 I said, it would have been the same data as release 19 management would provide. It would have been the data 20 around what PinICLs had been raised, were open and what 21 had been closed. 22 Q. Bearing in mind the references we saw in the risk 23 assessment -- the risk register to PinICLs being high -- 24 A. Yes. 25 Q. -- and to the concerns that were raised with you about 125 1 the increasing number of PinICLs in CI4, is it fair to 2 assume that you are likely to have been reporting that 3 the PinICLs remained high at this stage? 4 A. If I had reported again at that stage -- I don't know 5 that I reported since the December position. I mean, 6 there's multiple versions of the product so that CI4 7 recommendation is for a later development stage of it, 8 as opposed to PinICLs from earlier stages, so they're 9 all going to start overlapping to a certain extent. 10 Q. But is it your overall recollection PinICLs remained 11 high at this time, in the spring of 2000? 12 A. I think volumes of PinICLs continued for quite a while. 13 Q. By volumes, you mean? 14 A. Numbers being raised. 15 Q. High volumes? 16 A. Yes. 17 Q. The final entry we have is dated 10 May. This records 18 a response received from Mike Coombs, it reads: 19 "As discussed this should be closed. Effectively as 20 a management team we have accepted the ongoing cost of 21 maintenance rather than the cost of a rewrite. Rewrites 22 of the product will only be considered if we need to 23 reopen the code to introduce significant changes in 24 functionality. We will continue to monitor the code 25 quality (based on product defects) as we progress 126 1 through the final passes of testing and the introduction 2 of the modified CI4 codeset into live usage in the 3 network. PJ [that's presumably a reference to you, 4 Mr Jeram] can we make sure that this is specifically 5 covered in our reviews of the B&TC test cycles." 6 And the action is recorded as closed. 7 On 10 May. Just pausing there, the reference to the 8 cost of maintenance -- and we have already discussed 9 what we understand maintenance to be -- but it is 10 effectively the cost of continuing to rectify bugs, 11 errors and defects -- 12 A. Yes. 13 Q. -- in the live estate and in testing. 14 Can you explain, please, the reference at the very 15 end to making sure that this is covered in our reviews 16 of B&TC test cycles? 17 A. Yes, B&TC is the -- at this stage it went through 18 development testing into system testing and then B&TC, 19 which I think was something like business and technical 20 conformance test, something like that. So what this is 21 ensuring is that the reviews are that, that we're 22 monitoring what the PinICL position is coming out of 23 those test cycles. 24 Q. So, as at 10 May, the position has ultimately been taken 25 not to redesign or rewrite the EPOSS code; that's 127 1 correct? 2 A. Yes. 3 Q. It was a decision of the management team. As release 4 project manager were you part of that team? 5 A. I don't know what the team here would be, but if it's 6 the management team, no; but if it's a conversation 7 about what we're doing with development then I would 8 have probably joined that conversation with Terry and 9 Mike. 10 Q. Do you recall having input into the decision that was 11 ultimately taken in relation to the closure of this 12 action? 13 A. No. 14 Q. Now, we know that it had been brought to your attention 15 that there were increasing numbers of PinICLs, cash 16 account, misbalances and reconciliations in the CI4 17 testing. 18 A. Yes. 19 Q. Were those matters which you had brought to the 20 attention of your senior managers at the time that this 21 decision was taken? 22 A. Yes, it would have all been -- it would have been known. 23 Q. When you say "it would have been known"? 24 A. Well, because they would have known about what's 25 happening with CI4 at this point. If there were 128 1 problems in the testing at the beginning of April then 2 it would have been known by Terry and Mike. 3 Q. So, at the time of taking this decision, it's your 4 evidence that Terry Austin and Mike Coombs would have 5 been well aware of those concerns which had been brought 6 to your attention? 7 A. They would have been known -- well, I don't know what it 8 was like at the end of -- or the beginning of May, 9 a month after Duncan raised that email, I don't know 10 what happened during that month, whether after the 11 reviews, et cetera, how it improved. But that's 12 probably why they're saying "Let's monitor this in the 13 B&TC". 14 Q. This decision was taken, notwithstanding the fact that 15 there were serious concerns about ICL Pathway's ability 16 effectively to maintain Horizon in the live estate? 17 A. There were concerns raised in the risk register, yes. 18 Q. Did you consider at the time that this was the right 19 decision to take? 20 A. I would say yes because, if I didn't or hadn't have, 21 then I would have raised my voice to make a point, so 22 I must have agreed. 23 Q. Were you not concerned that the continued application of 24 software fixes was likely to lead to a further 25 degradation in the quality of the EPOSS code? 129 1 A. There was still a very large test team in place that 2 were validating things and continually -- well, that's 3 where the PinICLs were coming from, so continually 4 checking and validating and improving the product, so it 5 wasn't as if that exercise had stopped. 6 Q. You described that as an improvement of the product, but 7 you were aware, were you not, of a risk of what we call 8 code regression? 9 A. There's always that risk, yes. 10 Q. What did you understand by that risk? 11 A. That in changing a product -- in an ideal world, once 12 you put something into a live estate this never happens, 13 you would never change it, right? Whenever you change 14 it there is a chance that you will have a code 15 regression in it, that's why you continue your testing. 16 Q. So it doesn't follow necessarily that, albeit you might 17 improve certain aspects of the software, you might cause 18 yet further problems to arise elsewhere? 19 A. Yes and that's why you do regression testing. 20 Q. Was the likely consequence of code degradation caused by 21 further software fixes, was that not likely to cause yet 22 further problems, such as cash account imbalances and 23 reconciliation errors? 24 A. There is that risk but that's why you have the 25 regression testing. 130 1 Q. Having taken the decision not to redesign and rewrite 2 the EPOSS code, ICL Pathway continued to apply fixes to 3 the code as and when they were detected, did they not? 4 A. Yes, or as and when they were enhancing it for other 5 reasons. 6 Q. Do you recall whether the number of PinICLs and fixes 7 being applied in the summer and autumn of 2000 remained 8 high or not? 9 A. No, sorry, I don't. 10 Q. We have a copy of the release note for the Core System 11 Release Plus, which was produced in October 2000 and to 12 which you appear to have contributed; is that right? 13 A. Mm-hm. 14 Q. This document bears the reference FUJ00119319, please. 15 This document is dated 24 October 2000. It's the 16 version 1 of release note for the Core System Release 17 Plus. It provides a definition of the Core System 18 Release Plus for Post Office Counters Limited. Can you 19 please just briefly explain the purpose of a release 20 note of this type? 21 A. It's to record and share the contents of the release, so 22 whether that is change requests or PinICLs. CSR+ would 23 have been introducing new functionality, I'm sure. 24 Q. At appendix B, the note contains a known problem 25 register in which known problems in the release and any 131 1 fixes which have been applied were documented; is that 2 correct? 3 A. I'm sure it's correct. 4 Q. You have been provided with a copy of this document, 5 have you not? 6 A. Yes, I just don't know what particular part -- I can 7 look at appendix B, if you like, to answer your 8 question? 9 Q. No, not at all. We will pull it up. It's at page 34, 10 please. This is what's known as the known problem 11 register. 12 A. Yes. 13 Q. It runs from page 34 to page 39. I'm sure you can help 14 us but I think we can see the specific PinICL references 15 in the far left-hand column. 16 A. Mm-hm. 17 Q. So their reference numbers -- 18 A. Yes. 19 Q. -- and a short summary description in the next column. 20 Then ICL Pathway's business impact assessment of the 21 PinICL; is that right? 22 A. Yes. 23 Q. Can you explain the grading, please, for that? We see 24 A, B and C. 25 A. I don't know, but I can give you a view. I think A 132 1 would be that it was a major impact, B of lesser impact 2 and C a low impact. 3 Q. We then see in the next column the business impact on 4 the Post Office Network. We see a record of its status 5 on 10 October and, in the final column, we see 6 commentaries, including whether or not a fix had been 7 applied, whether a fault has been found -- 8 A. Yes. 9 Q. -- you follow that? 10 A. Yes. 11 Q. We, of course, don't know a great deal about each and 12 every one of these PinICLs and their summary 13 descriptions won't assist us in that regard, but we can 14 see several relating to the EPOSS -- we see quite 15 a number relating to the EPOSS product, or identified as 16 such. Do you accept that, having reviewed this 17 document? 18 A. Yes. 19 Q. If we could scroll down a little bit, please. Some of 20 these are rated A, so very serious. We see 41673, the 21 "CSR+/EPOSS: OW Sales report negative volume", 22 categorised A by ICL Pathway and we can see it has been 23 closed, a fix having been applied. 24 If we go on to the next page, please, we can see 25 a further PinICL at 45573, relating to "Office Balancing 133 1 Barred". Presumably, this is another issue in EPOSS if 2 it's to do with office balancing; is that a fair 3 inference? 4 A. These, of course, could be problems introduced into CSR+ 5 code and then found during the testing of the CSR+ code. 6 Q. Indeed, in theory, I'm not suggesting -- because, of 7 course, this is a release note. This is prior to the 8 release of CSR+ into the live estate -- 9 A. Yes. 10 Q. -- is that correct? 11 A. Yes. 12 Q. So PinICLs discovered during testing, software fixes 13 have been applied and, on the basis of that, the 14 software is deemed to be fit to be released into the 15 live estate? 16 A. Well, that would be the discussion. Whether the PinICLs 17 that aren't fixed, whether that's an acceptable position 18 to go into live. 19 Q. So there are a number of others. I won't take you 20 through them all, but 47132, the PinICL is "Cannot 21 transfer existing transaction", again graded A by 22 ICL Pathway, medium severity by Post Office Counters 23 Limited, and closed following another PinICL fix. 24 Could we scroll down a little bit further, please. 25 Two further PinICLs relating to EPOSS and graded high at 134 1 48796 and 488 -- 2 A. I think the concern would be if there was an A here that 3 wasn't fixed. 4 Q. Well, I think that's precisely the question, isn't it, 5 because, of course, by repeatedly applying these 6 software fixes, you were creating a risk of generating 7 yet further faults and defects in the code, were you 8 not? 9 A. I think that would depend -- well, I mean CSR+ would be 10 enhancing the CSR product and bringing in new 11 functionality that the Post Office wanted, so these 12 issues could be in the new functionality in the EPOSS 13 example that's come in with EPOSS, as opposed to the old 14 product, if you like, the product from CSR. 15 So, I mean, the problems could come from the 16 development work done for CSR+. I think you're saying: 17 is that because it wasn't a stable product in the first 18 place? I don't think you can draw that necessarily from 19 this. 20 Q. But we do know that, from the emails we have seen, there 21 were quite significant concerns about the number of 22 PinICLs being raised in CI4 -- 23 A. CI4, yes. 24 Q. -- or what we believe is one of the releases connected 25 with CSR+? 135 1 A. Yes, I don't know the position of CI4 as against CSR+. 2 Q. Looking back, knowing what we know now, do you consider 3 that deciding not to redesign and rewrite EPOSS, in the 4 face of the advice of ICL Pathway's technical experts, 5 was the right decision to have made? 6 A. That's a very difficult one to answer. Would there have 7 been less problems and less work if it had been 8 redesigned or redeveloped, or would it have created its 9 own problems by starting again? That would be a risk as 10 well and I'm sure that's the kind of decision that 11 people were making because to start from scratch might 12 introduce its own problems. I think the view at the 13 time was that the problems were in certain areas, as 14 opposed to generally across EPOSS and, therefore, they 15 would concentrate on those areas. 16 Q. Why did you consider that starting again might simply 17 introduce more problems? What led you to that 18 conclusion? 19 A. It is just the risk of going and redeveloping something 20 right from scratch again. 21 Q. Presumably the purpose of doing that was to ensure that 22 it was done correctly -- 23 A. Yes. 24 Q. -- the next time. 25 A. Which I presume people thought they were doing the first 136 1 time, but yes. 2 Q. But that didn't seem to be borne out by the very high 3 number of PinICLs and the advice of the technical 4 experts -- 5 A. Yes. 6 Q. -- at the time, was it? 7 A. I think there were two views, I believe, and the 8 decision was made assessing those two views. 9 Q. Can you help us, what do you mean by "there were two 10 views"? 11 A. Because I believe some people thought that it could be 12 fixed in the areas that needed fixing and improving and 13 others that felt that the whole thing should maybe be 14 rewritten. 15 Q. You have concluded to yet another view, which is that, 16 by rewriting it, you might create just as many problems. 17 From where did that view stem? Can you help us? 18 A. It's just a sort of feeling I got from the conversations 19 at the time. When Terry was leading on deciding which 20 way to go, those were the kind of decisions or 21 discussions that were taking place. 22 Q. I wonder, Mr Jeram, if we could return, please, to your 23 witness statement at WITN04180100. 24 SIR WYN WILLIAMS: At some stage, Ms Hodge, there's probably 25 a need for a break, I'm guessing, but you choose your 137 1 moment, all right. 2 MS HODGE: Sir, I have two more questions for the witness. 3 SIR WYN WILLIAMS: Oh, well, I'm sorry, carry on, please. 4 MS HODGE: I propose we take a break at that stage and -- 5 SIR WYN WILLIAMS: Yes, of course. 6 MS HODGE: -- then permit others to ask questions. 7 Mr Jeram, you have taken us to your evidence at 8 paragraph 15 on page 4, please. I think you accept that 9 this paragraph does not give a full account of what you 10 knew at the time of issues relating to EPOSS and the 11 cash account; is that right? 12 A. Yes. 13 Q. I wonder if you could please explain to me why that is. 14 A. I had forgotten my involvement in that area. 15 Q. I asked you a short time ago about your reference to 16 formal audits at paragraph 26. Do you recall that? 17 A. Yes. 18 Q. You said that you asked to be shown some of those 19 audits? 20 A. Yes. 21 Q. Was the CSR+ development audit one of the audits which 22 was shown to you at the time you prepared this 23 statement? 24 A. I don't think it was. I might be wrong. I don't think 25 it was. There was a warehouse audit and something else. 138 1 I don't remember, sorry. 2 Q. Looking at what you said at paragraph 26 on page 8, 3 under the heading "Fitness for Purpose", your statement 4 reads: 5 "ICL Pathway continually reviewed its work to make 6 improvements for future releases, this would have 7 included formal internal audits, although I do not 8 recall any of these audits specifically. Internal 9 auditing prior to the national rollout was owned by 10 Martyn Bennett, Head of Quality Management, and by his 11 responsible Internal Audit Manager, Jan Holmes." 12 This doesn't, does it, give a true and fair 13 reflection of your knowledge at the time from the 14 internal audits which were shown to you, does it? 15 A. At the time -- 16 Q. Forgive me, "at the time" being at the time of your 17 involvement in the Horizon project. 18 A. True. So I knew about the audits when I was on Horizon, 19 yes. This is saying I didn't recall any of them in 20 remembering them as part of the statement, yes? 21 Q. What you are saying, in effect, is that you simply had 22 no recollection at the time of writing your witness 23 statement of the very serious concerns that were raised 24 about the accounting integrity and the quality of code 25 and the maintainability of the Horizon System. 139 1 A. Yes. 2 Q. Is that your evidence? 3 A. Yes. 4 MS HODGE: Thank you. I have no further questions. 5 SIR WYN WILLIAMS: Right. Sorry for the interruption. So 6 do we have questions from other legal representatives? 7 MR JACOBS: Sir, I have one question. It's probably going 8 to take about three minutes, if that assists. 9 SIR WYN WILLIAMS: And Ms Page? 10 MS PAGE: There may be slightly more than that from me, if 11 I can just have a few minutes to have a look at my -- 12 SIR WYN WILLIAMS: Well, I will tell you -- 13 Can I address you, Mr Jeram. Would you prefer to 14 have a short break now and allow everybody to gather 15 their thoughts, or would you prefer to carry on until 16 the end, on the assumption that the end is no more than 17 about ten minutes away? 18 A. I'm going to ask Ms Page. I think she would prefer me 19 to have a break so -- no? 20 MS PAGE: No, that's fine. 21 A. Then can we continue, please. 22 SIR WYN WILLIAMS: Fine. Let's go to the end then. 23 Questioned by MR JACOBS 24 MR JACOBS: Good afternoon, Mr Jeram. I ask questions on 25 behalf of 153 Core Participants who are subpostmasters 140 1 and I am instructed by Howe & Co. 2 Can we turn to paragraph 16 of your statement. 3 That's WITN04180100. This is paragraph 16, which is on 4 page 4 of 9. This is in relation to the cash account 5 and you say that, prior to Horizon, subpostmasters used 6 a paper accounting system. You say that Post Office 7 took a decision which wasn't taken quickly that there 8 should be no paper cash account; is that right? 9 A. Yes. 10 Q. Now, the effect of this decision is that subpostmasters 11 were prevented from checking their records against 12 allegations of shortfalls. They didn't have the paper 13 system -- 14 A. Yes. 15 Q. -- and the Horizon System didn't permit them to do that. 16 The question I have for you is: was there any discussion 17 between you, as development and later programme 18 director, and Post Office on this issue? 19 A. No. 20 Q. Were you aware of the issue at the time? 21 A. The issue of? 22 Q. That the ability of subpostmasters to have records that 23 they could check and interrogate was going to be taken 24 away from them in the Horizon System? 25 A. No. 141 1 Q. I'm asking you these questions because you have referred 2 to it in your statement. 3 A. Of course. 4 Q. The final question then is: do you agree from what you 5 have said about the Post Office decision-making process, 6 and that this decision was a decision that Post Office 7 made, that the ability of subpostmasters to check their 8 records was deliberately designed out of the Horizon 9 System? 10 A. Yes. Obviously moving to an automated system at some 11 point, you would move away from the paper side and 12 I think the papers had to be sent in to TIP and TIP had 13 to process them and -- or whatever, but yes, that 14 decision took that away. 15 MR JACOBS: Okay. Well, thank you. I don't have any 16 further questions for you, unless I'm asked to ask you 17 anything else. 18 No, I'm not. Thank you. 19 Questioned by MS PAGE 20 MS PAGE: Flora Page, representing a number of the 21 subpostmasters. 22 You became programme director in 2001? 23 A. Yes, the end of 2001 when Mike Coombs unfortunately was 24 taken ill. 25 Q. So presumably that meant there wasn't much of 142 1 a crossover. 2 A. No. 3 Q. But you knew the programme pretty well already, 4 didn't you? 5 A. Yes. 6 Q. And when you took over, what was the line of report 7 between you and the SSC? 8 A. SSC? 9 Q. Yes. 10 A. That's -- 11 Q. Third line support. 12 A. -- service management. 13 Q. Sorry? 14 A. In service management? 15 Q. Third line support. 16 A. Okay. None. 17 Q. So how would you have had any control over what they 18 did? 19 A. I wouldn't have had. 20 Q. You wouldn't have had? Why not? 21 A. Because they're managing the live estate. The programme 22 is managing future change, not what's in live estate, so 23 I don't think service management reported to the 24 programme director. 25 Q. So how would problems in the live estate get 143 1 communicated to your programme? 2 A. Through the raising of PinICLs. 3 Q. And how would your programme come to know about PinICLs? 4 A. So they would have been routed -- my team, development 5 team, would have been fourth line, so when there was 6 something that was felt by the second and third that it 7 required an investigation and maybe a change into 8 software, then that would be routed through to the 9 development teams. 10 Q. So the fourth line were under your command, as it were? 11 A. Yes. 12 Q. Who was that person, who was the link? Who would have 13 reported to you from fourth line? 14 A. Okay, when I was programme director -- so the 15 development director would have managed the development 16 teams and there would have been a number of those 17 different teams and they would have had resolution 18 groups, so the PinICLs would have been sent to those 19 different resolution groups. 20 Q. Right, but how would you have ensured that problems that 21 were arising in the live estate didn't continue into the 22 future programme? 23 A. So following the route, let's say the problem raised 24 through live through Post Office, investigation from 25 third line believes it's a software fault, that goes to 144 1 a fourth line team to resolve, which would have been 2 a resolution into the live estate and we used to use 3 something called a clone. They would then clone that 4 PinICL, a copy of that PinICL, into the version of 5 software they were then working on for the next release. 6 So the fix would be applied into both. 7 Q. You knew, didn't you, that throughout the year 2000 8 Acceptance Incident 376 had been a live issue, something 9 that needed to continue to be monitored? 10 A. Yes. 11 Q. So what did you do to ensure that the programme going 12 forward would be alert to and able to continue to 13 monitor what was going on with cash accounts, 14 reconciliations, AI376 generally? 15 A. I think that would have been through the incidents that 16 came from the live service. 17 Q. So the route that you have just described? 18 A. Yes. 19 Q. Are you aware of that working? I mean can you be sure 20 that AI376 continued to be monitored and fed through to 21 your team for the future? 22 A. I believe the process worked. 23 Q. What makes you say that? 24 A. I mean, if there were bigger problems than just a flow 25 of PinICLs in the live estate I would expect that to 145 1 have been brought to the attention of the management 2 team that I was then part of, via the service director. 3 Q. Given what we know now and the fact that there were 4 continuing problems, how can you be confident that this 5 was working? 6 A. I don't know how I can be confident. That was the 7 process we had in place that we believed worked. 8 Q. Were you in any way involved with making sure that if 9 POCL wanted to pursue postmasters, audit trails were 10 made available to them? 11 A. No. 12 Q. In 2001 when you took over, who do you think would have 13 been? 14 A. It would have been managed by the service group. 15 Q. And who was that? 16 A. They were under Steve Muchow. 17 MS PAGE: Thank you. 18 SIR WYN WILLIAMS: Is that it, Ms Page? 19 MS PAGE: Those are my questions, thank you. 20 SIR WYN WILLIAMS: That's it, Ms Hodge? 21 MS HODGE: That's right. Thank you, sir. The witness can 22 be released. 23 SIR WYN WILLIAMS: So thank you very much, Mr Jeram, for 24 coming to the Inquiry to answer all the questions put to 25 you. 146 1 As I have said, we will now adjourn these hearings 2 until 10.00 on Tuesday. 3 MS HODGE: Thank you, sir. Good afternoon. 4 A. Thank you. 5 (3.10 pm) 6 (The Inquiry adjourned until 10.00 am on Tuesday, 7 15 November 2022) 8 INDEX 9 10 MARK JAROSZ (sworn) ..................................1 11 Questioned by MR BLAKE ...........................1 12 Questioned by MR JACOBS .........................68 13 Questioned by MS PAGE ...........................76 14 MR PETER JERAM (sworn) ..............................87 15 Questioned by MS HODGE ..........................87 16 Questioned by MR JACOBS ........................140 17 Questioned by MS PAGE ..........................142 18 19 20 21 22 23 24 25 147