1 Thursday, 9 March 2023 2 (9.58 am) 3 MR BEER: Sir, good morning can you see and hear me? 4 SIR WYN WILLIAMS: Yes, thank you very much. 5 MR BEER: Can you see Mr Roll? 6 SIR WYN WILLIAMS: Yes, I can. 7 MR BEER: May I therefore call Richard Roll, please. 8 SIR WYN WILLIAMS: Yes. 9 RICHARD WILLIAM ROLL (affirmed) 10 Questioned by MR BEER 11 MR BEER: Good morning, Mr Roll. Can you see and hear me 12 clearly? 13 A. Yes, thank you. 14 Q. As you know, my name is Jason Beer and I ask questions 15 on behalf of the Inquiry. Can you give us your full 16 name, please? 17 A. Richard William Roll. 18 Q. Can I extend the Inquiry's thanks for the provision of 19 a witness statement by you and for attending to give 20 evidence remotely today. You should have in front of 21 you a pack of hard copy documents, the first of which, 22 at tab A, is a witness statement you made on 2 February 23 this year. It's 18 pages long. Can I invite you to 24 open that and confirm that that is your witness 25 statement. 1 1 A. Yes, it is. 2 Q. For the Inquiry's reference, the URN for that is 3 WITN00780100. Can you turn to the 15th page of it, 4 please. 5 A. Yes. 6 Q. Is that your signature? 7 A. It is. 8 Q. Then if you turn back to the first page, to the first 9 line, where it says, "I joined the RAF in 1974", should 10 that read "1976"? 11 A. It should. 12 Q. Other than that correction, are the contents of the 13 witness statement true to the best of your knowledge and 14 belief. 15 A. Yes. 16 Q. Thank you. I am going to ask you some questions today 17 about issues that the Inquiry has grouped together in 18 what we call Phase 3 of the Inquiry, namely your role in 19 the operation of the Horizon System and the work of the 20 SSC, which is variously described in Fujitsu documents 21 and the Post Office as the System Service Centre, the 22 System Support Centre or the Software Support Centre. 23 They're all referring to the same thing and I'm going to 24 call it the SSC today. You understand? 25 A. Yes. 2 1 Q. I'm not going to ask you questions about what the Post 2 Office did in relation to and in response to your 3 appearance on Panorama in 2015, nor am I going to ask 4 you questions about the basis of many of the suggestions 5 that were put to you over the course of a day and 6 a half's cross-examination on 13 and 14 March 2019 when 7 you appeared as a witness in the Bates v Post Office 8 trial in the High Court, just down the road from here, 9 and nor am I going to ask you questions about the 10 treatment more generally of your evidence by the Post 11 Office in the trial, nor the conclusions that Mr Justice 12 Fraser drew about the accuracy and reliable of your 13 evidence. Do you understand? 14 A. Yes. 15 Q. All of those issues or some of them may be examined 16 later in the Inquiry but we do not need your evidence in 17 order to examine them. So can I start, then, with your 18 background and experience. As we've just established, 19 I think, you joined the RAF in 1976; is that right? 20 A. That's right. 21 Q. In one of your statements prepared for the Bates 22 litigation, you say that your title in the RAF was 23 avionics engineer; is that right? 24 A. Yes. 25 Q. You tell us in that statement that you worked on 3 1 a variety of mainframe computer systems and that you 2 were selected for a software development team working on 3 aircraft control and attack systems; is that right? 4 A. Broadly speaking, yes. 5 Q. I think it's right that, in your time in the RAF, you 6 obtained a City & Guilds Level 3 certificate in 7 electronics? 8 A. Yes. 9 Q. You obtained an ONC in electronics engineering -- 10 A. Yes. 11 Q. -- and you obtained an HNC in software engineering? 12 A. I did all of the modules for that and got distinctions 13 and credits but I never completed the final module so 14 I never actually obtained the final HNC. 15 Q. I understand, thank you for that clarification. Then 16 after 14 years or so in the RAF in 1989 you left; is 17 that right? 18 A. That's right. 19 Q. Over the next 12 years or so you undertook a range of 20 work, including, I think, bringing up some children on 21 your own before joining Fujitsu in January 2001? 22 A. Yes. 23 Q. Can we just look at page 17 of your witness statement. 24 It will come up on the screen or you can follow in the 25 hard copy. Page 17, please. Ah. That seems to have 4 1 been redacted. Is it redacted in the copy that you've 2 got? 3 A. Um, which page am I looking at? Page 17 on my witness 4 statement? 5 Q. Yes, the second page of your CV. 6 A. Oh, right, yes. I've got it in front of me here. 7 Q. You've got it in front of you? 8 A. Yes. 9 Q. Okay. I think you, working from the bottom upwards -- 10 we can take that down from the screen, thank you -- 11 working from the bottom upwards, the first job after the 12 RAF, was that working in robotics, essentially? 13 A. Yes. 14 Q. Then the job above that, between August 1990 and March 15 1991, was that software development? 16 A. Software support to development yes, we were rolling out 17 a product in the UK and some of it had to be modified as 18 it went along for the UK customers. 19 Q. Then May '91 to December '92, business process analyst. 20 What was that? 21 A. The company, new information paradigms, it was 22 pre-Internet. They had a product which could 23 interrogate databases, such as Reuters, some of the 24 online financial databases, et cetera. It could access 25 those overnight, download the information, format it, 5 1 and print a document so that in the morning you would 2 have an up-to-date management report on what the 3 competitors were doing, et cetera. At the time, it was 4 all cutting edge stuff. It was written in language very 5 much like HTML is today but, as I say, predated the 6 Internet by several years. 7 Q. Thank you. Then for two and a half years, as I've said, 8 you worked looking after your children? 9 A. Yes, I tended to do odd jobs for -- I worked for the 10 Natural History Museum on a database project in the 11 evenings from home. I took the children to school 12 during the day, picked them up from school, get them to 13 bed and then I'd work until 2.00 in the morning or so on 14 the database projects and then get some sleep, get them 15 to school in the morning, couple more hours sleep, get 16 up at lunchtime, do the housework, et cetera -- 17 Q. I see. 18 A. -- pick up the kids. Then. 19 Q. Then between June '95 and July '95 you were a desktop 20 implementation engineer, what does that mean? 21 A. That was the title. There was a company called TAL, 22 again it was really when IT was still taking off, as it 23 were. It had been set up by a previous employee at 24 Glaxo to manage or help manage Glaxo's IT systems. He 25 then contracted into them, if you like. So we were 6 1 responsible for configuring desktop computers, 2 installing them, building them from scratch in some 3 cases, training people on the software that was being 4 used on them, so on and so forth. 5 Q. Then for the same company you worked as a systems 6 procurement analyst. Can you explain what that is, 7 please? 8 A. Yeah, the -- they then needed someone again, through TAL 9 to work. There was a problem on the Glaxo recession 10 development site and they weren't getting the equipment 11 they needed and the software and hardware they needed to 12 be able to process the data, get the drugs to market, 13 basically. They needed someone else there to help speed 14 up the process. 15 So I was asked to step in and help with purchasing, 16 analysing what equipment they needed, what desktops, 17 what processing power was best for their needs. So then 18 I would then source the -- source the equipment and get 19 the purchasing done to get it onto the desks as quickly 20 as possible. 21 Q. Then after that, between January and December 1996 you 22 worked in the same company again as a project manager; 23 is that right? 24 A. Yes, they -- Glaxo merged with Wellcome at that time, so 25 there was a lot of staff. At Dartford we had a lot of 7 1 data, a lot of systems, they needed bringing in line 2 with Glaxo equipment. Some of the staff were made 3 redundant, the rest were being transferred to another 4 site. 5 Q. Then towards the top of the page there two jobs, firstly 6 as a systems manager and then as a project manager. 7 A. Yes. 8 Q. Did any of that involve work with software, or writing 9 software or coding? 10 A. The job with at CRO Catalyst, I was responsible for 11 looking after all the software in the UK and Europe, so 12 that involved configuring software on the servers in the 13 Hague, also in Switzerland and the UK. That was more 14 setting up software rather than coding or writing it. 15 Q. Thank you. Then at the foot of the preceding page, in 16 early '99 and then for the rest of '99 and to the middle 17 of June 2000, a configuration centre manager and then 18 a system support analyst. Can you describe what those 19 jobs were, please? 20 A. Yeah, sure. The configuration centre manager, Bitech 21 had a large facility in Bracknell, configuring IBM 22 minicomputers, setting up software, et cetera. They 23 were moving the whole process to Germany and closing 24 down the UK facility. My manager in the UK had moved 25 out to Germany and they needed someone else to step in 8 1 while they closed the rest of the facility down in the 2 UK. So I took it over for that period, for a short 3 period of time, to run it whilst they moved most of the 4 stuff out -- 5 Q. And the system support analyst? 6 A. That was running a third, if you like, of the global 7 sales database software. I was responsible for managing 8 the software in Egypt, Greece, Tunisia, the 9 Middle Eastern areas, part of Africa, most of Europe, 10 most of Eastern Europe. That involved writing code -- 11 I'm trying to remember exactly what the terms were. 12 Basically, the sales reps would input the data in their 13 various countries. That would then be consolidated into 14 a database in the UK. That was an Oracle Database, 15 I think. 16 So I had to manage the Oracle Database, also write 17 the scripts to interrogate the database, so that the 18 sales reports were generated correctly. There were 19 often issues with data coming in from various countries 20 that would be out of sync, so that all had to be sorted 21 out, turn the database off, go in, sort the code out, 22 sort the actual records out, put it all back together, 23 and then resync it with the databases in Greece, 24 Romania, wherever. 25 Q. I understand. Thank you. Then I think you took up 9 1 employment for a period of three years and six months 2 between January 2001 and August 2004 with Fujitsu? 3 A. Yes. 4 Q. The job title that I've seen for you was IT product 5 specialist; is that right? 6 A. I think so, yes. 7 Q. You worked in third and fourth line support in the SSC; 8 is that right? 9 A. Yes. I think now that that -- it was probably classed 10 more as third line support. There was some development 11 but probably technically -- probably better described as 12 third line support. 13 Q. Thank you. Was your work focused almost exclusively, 14 therefore, on the investigation and resolution of issues 15 and problems with the Horizon System? 16 A. Yes. 17 Q. Did that involve you dealing directly with 18 subpostmasters and others employed in branches? 19 A. Yes. 20 Q. You were, I think, based at the Fujitsu offices in 21 Bracknell for the entirety of that period? 22 A. Yes. 23 Q. Can I ask about the size of the SSC team. In one of 24 your statements, you say that there were over 25 30 individuals working on the same floor as you in 10 1 Bracknell. By that, were you meaning they were all in 2 the SSC? 3 A. Not all of them. Some of them would have been in the 4 testing team. Probably -- I think there were 25 to 30 5 SSC members and half a dozen or more people in testing. 6 Q. Thank you. So that 25 to 30, were they all doing the 7 same or substantially the same job as you? 8 A. Substantially the same job as me, yes. 9 Q. Did you manage anyone? 10 A. No. 11 Q. How many managers were there of the SSC? 12 A. Just one, as far as I can recall. 13 Q. Who was the manager of the SSC? 14 A. Mik Peach. 15 Q. Did he have a deputy? 16 A. Um, Steve Parker stood in for him when he wasn't there, 17 yeah. 18 Q. What was the structure of the team? How were the 25 to 19 30 of you, other than Mr Parker and Mr Peach, arranged 20 or organised, if you can remember? 21 A. It was a very flat management structure. We just all 22 reported to Mik Peach. Physically on the floor, we had 23 own little desk space with two computers on it. One was 24 completely secure and that was connected to the Horizon 25 System, and the other one was an open system, for want 11 1 of a better way of putting it, where we could send 2 emails, look up things on the Internet if necessary. 3 That sort of thing. 4 Q. So a flat structure, all reporting in to Mr Peach, no 5 hierarchy within the 25 to 30 of you? 6 A. No, not that I remember. 7 Q. Was there any division in terms of specialism amongst 8 you, in terms of the work that was undertaken? 9 A. Um, yeah, some of the guys there had been working with 10 Unix systems since the year dot, so they were, you know, 11 real experts on Unix. So only knew if there was 12 a problem with the server farm, they would pick up those 13 problems as some of them were very, very good on the 14 financial side of things, mathematics and that, so they 15 tended to pick up any work that came on, and that sort 16 of thing. Some of us were just sort of generalists who 17 would dive in and do anything we could and, if we got 18 stuck, then we knew who we'd perhaps go and ask for 19 a bit of help. 20 Q. Thank you very much. I want to ask how you came to give 21 evidence and to speak out about the Horizon System. 22 I think it's right that you came forwards after seeing 23 the BBC South Inside Out investigation that was 24 broadcast in, I think, 2011; is that right? 25 A. I can't really remember. There was something I saw or 12 1 read and it just triggered some memories, and I just 2 knew that we'd been busily trying to patch systems 3 behind the scenes and it seemed wrong that -- well, it 4 might have been wrong that postmasters may have been 5 getting the blame for something that actually wasn't 6 their fault. 7 So I just contacted someone, I'm not sure who, and 8 said, "I used to work on the systems", and if they 9 wanted to talk to me, you know, I'd be willing to have 10 a chat and explain what we did. 11 Q. So what was it that triggered you coming forwards? What 12 did you see or read? You mentioned there, I think, 13 postmasters getting the blame. In what way were they 14 getting the blame? 15 A. Being sent to prison or prosecuted for things that 16 weren't necessarily their fault. It seemed 17 an injustice. 18 Q. So did you essentially become a whistleblower? 19 A. Yes, I didn't think of that term at the time until it 20 was mentioned, you know, years later, but yes. 21 Q. Did you speak, give an interview, to Panorama in 2015? 22 A. Yes. 23 Q. As I've said already, you gave evidence before 24 Mr Justice Fraser in the Group Litigation Order 25 proceedings over a day and a half on 13 and 14 March 13 1 2019? 2 A. Yes. 3 Q. Can I ask you to look at your witness statement, please, 4 paragraphs 7, 8 and 9, which is on page 4 of the witness 5 statement. I'm going to explore here the nature of the 6 issues that were referred to you in the SSC. 7 Sorry, it's my mistake. Can we have up on the 8 screen POL00029991. 9 It's my mistake, Mr Roll, it was paragraphs 7, 8 and 10 9 of this document that's going come up on the screen 11 for you that I wanted you to look at, rather than your 12 Inquiry witness statement. This is a copy of the 13 witness statement -- if we just scroll up to the top of 14 it -- that you made in the High Court proceedings. It's 15 dated 11 July 2016 and it's the first of two witness 16 statements that you made, okay? 17 A. I think I made three witness statements. 18 Q. Ah, we've got two. We'll explore where the third one 19 has gone. 20 Can you see paragraph 7 at the foot of the page? 21 You're introducing the work that you did in the SSC. 22 A. Yes. 23 Q. You say: 24 "By way of example the type of issue that I would 25 deal with, if a financial discrepancy had arisen in 14 1 a branch (eg a 'shortfall' of £5,000) then I would need 2 to work sequentially through all transactions over the 3 relevant period, and also work through thousands of 4 lines of computer coding. Software programs were 5 written by us to strip out irrelevant data to enable us 6 to more easily locate the error." 7 I want to ask you some questions about that, please. 8 You say you would need to "look sequentially through all 9 the transactions over the relevant period", and why 10 would you have to do that? 11 A. If there was an error of -- I mentioned £5,000 there, 12 but quite often it would be a random, you know, 13 £4,011.27 or something. You would need to look at all 14 the transactions to see which one was at fault. If you 15 were lucky, you would find one for that exact value but, 16 more often than not, there wouldn't be one and it would 17 be a sum of several transactions, so you'd then be 18 trying to work out which transactions it was that, added 19 together, came up with that value. If you could easily 20 locate those values and those transactions, you would 21 then need to work out why that error had occurred, what 22 had gone wrong to cause the error. 23 Q. So just stopping there. How would you do that first 24 task, looking sequentially thorough all of the 25 transactions over the relevant period? 15 1 A. You would download the data from the database, for that 2 particular Post Office or counter, over the period of 3 perhaps 24 hours. 4 Q. How would you look through it? 5 A. Sorry, how would you look through it? 6 Q. Yes. Would you scroll or would you have something to 7 help you? 8 A. It varied. Sometimes you would scroll through the 9 pages, other times you'd print it all off. Using 10 various text editors and computer languages, we could 11 strip out all the irrelevant text so that would then 12 just leave the actual products and the values. So then 13 you could see what it was that they were selling there, 14 17 stamps at 49p each, or whatever. 15 Q. Sorry, Mr Roll, to interrupt you, just stopping you 16 there, you've moved to the bit at the end of the 17 sentence or the paragraph "Software programs were 18 written by us to strip out irrelevant data." 19 Who is the "us" in that sentence? Was that the SSC? 20 A. Yes. I wrote some myself. 21 Q. So you wrote software that had the purpose of removing 22 irrelevant data lines or data from the data that you 23 were looking at, so that you could try and focus on the 24 discrepancy in issue? 25 A. Yes. 16 1 Q. Could you give us an example of how such software might 2 strip out irrelevant data? 3 A. That's very difficult to explain without demonstrating 4 it or without showing you what the code looked like. If 5 you're familiar with what HTML code looks like, with the 6 angle brackets and the different tags, you can imagine 7 that there are lines and lines of code with that sort of 8 data in it. You may only have had four lines -- 9 sometimes you might only have one line that actually had 10 any data that was relevant that you could actually read. 11 So we would write a program that would -- it would 12 pass the text, source text, line by line, and if it 13 found any of the relevant code -- relevant tags that we 14 didn't need, it would then strip those and it would then 15 write the -- anything that was relevant into a text 16 file. And then that text file would then be a clean 17 text file which we could actually read physically, much 18 more readable, in a list. We could do the reverse as 19 well. We could correct data and then, using a program, 20 put all the tags back in to then put it back into 21 a database. Does that explain suitably what I'm talking 22 about? 23 Q. Yes, it does. Thank you. You say in this paragraph 24 that you would also work through thousands of lines of 25 computer coding. Why would you be looking at the 17 1 computer coding? 2 A. At times we were asked to try to identify -- we could 3 perhaps identify where an error had occurred in the 4 data, which lines of work it was. So then at times we 5 were asked to look at the source code for Horizon and 6 try and work out what exactly was going on in the source 7 code that caused that problem. We could then give it 8 back to the developers and say, "Here's the problem, 9 this is the source code, this is the source line, it's 10 wrong. It says here minus this value when it should say 11 plus this value", or whatever. 12 Q. Thank you, what would give you clue to thinking there 13 was something wrong with the source code and therefore 14 you would be examining the source code, the computer 15 coding? 16 A. Well, if you were going through the figures and you 17 could see quite plainly that they were maybe selling 18 stock and but one of the stock items, rather than the 19 money coming into the till, had actually been debited 20 from the till, then you'd think "Well, why is it doing 21 that? Why is the software saying it's been taken out 22 when, actually, it's come in?" So you might have 23 something like that and that's when you'd be able to go 24 to the code and think "Well, okay, where is it? What's 25 going on here?" 18 1 Q. So you would track the issue back into the code? 2 A. Sometimes, yes. 3 Q. In the example you've given, would that be visible or 4 apparent to the subpostmaster at all? 5 A. Not necessarily. Sometimes the errors might only crop 6 up when the data was actually being processed on the 7 overnight batch processing, from what I remember. I'm 8 a bit hazy around this now. 9 Q. If we carry on into paragraph 8 of this statement, you 10 say: 11 "If there was a single error then that would be easy 12 to identify, however there were often multiple errors 13 which would 'snowball'." 14 A. Yeah, that's what I was trying to explain a minute ago, 15 where, if you've got that one value and that jumps out 16 at you, then it's quite easy to spot. But if you've got 17 several items that are being added incorrectly or 18 whatever, dealt with incorrectly, then it could be very 19 difficult to work out exactly which items or which 20 products were causing the problem. 21 Q. In that sentence there, are you referring to errors of 22 calculation or errors in the code or both? 23 A. It could be either. Although, generally, the code 24 caused the errors in the calculation at some point. 25 Q. How obvious was a single error in Horizon coding? 19 1 A. Um, sometimes, from what I remember, quite easy to spot, 2 and other times we couldn't find -- we couldn't work out 3 what was going on. 4 Q. You say there were often multiple errors and, as you've 5 explained, that could mean multiple errors of coding 6 which would snowball and that this would make matters 7 more complicated. 8 Where -- sorry, Mr Roll, do go on. 9 A. Multiple errors, it's difficult to say whether it was 10 multiple errors in the coding or just one error that was 11 having multiple effects on the accounts. 12 Q. When you identified an error in the Horizon coding or 13 some data corruption, could you tell how and when the 14 error had been made? 15 A. Sometimes. 16 Q. What would delineate when you sometimes could and 17 sometimes couldn't? 18 A. There was -- if it was one of -- a particular 19 transaction on the counter, so that counter software was 20 at fault, then the -- there would be a time stamp in the 21 database, which you could use to give you a time when 22 things had gone wrong. But that's about all I can 23 remember from that. 24 Q. Would you be able to tell whether it was an error in the 25 original writing of the code or an error which had been 20 1 introduced by some other coding within Horizon? 2 A. No, not necessarily. 3 Q. Was a primary aim of you and your team not just to 4 identify the error in coding or data corruption but also 5 to ensure that they were fixed? 6 A. Our primary aim was to keep the system up and running so 7 that it worked and so that Fujitsu didn't suffer any 8 penalties, or the -- all the transactions had to go 9 through within the three-day limit. If we could 10 identify problems in the coding as we went along, then 11 that was a bonus. 12 Q. So is that why you described it as "patching it" earlier 13 on? 14 A. Sorry, as "patching it"? 15 Q. Patching it up as you went along? 16 A. Yeah, we were, yeah. We were patching the system as 17 a whole, not necessarily the code. 18 Q. You mention there that you understood that Fujitsu would 19 suffer financial penalties, I think, in the event of 20 delays in processing; is that right? 21 A. Yes. 22 Q. What was your understanding of those? 23 A. It's a long time ago and I can't remember the figures 24 exactly. My understanding was that if, for instance, 25 a bank transfer didn't go through within three days, 21 1 I think it was, then there would be a financial penalty 2 of -- I can't remember, I think it was -- I don't know 3 whether it was 10 pence or £10. It was a smallish 4 financial penalty. 5 The issue arose when you've got 20,000 counters or 6 20,000 post offices, maybe 40,000 counters, whatever, 7 sending the data through overnight for processing, so 8 then that small financial penalty is multiplied 9 thousands and thousands of times by the number of 10 transactions that are being held up. So then, the SLAs 11 that we were trying to meet could have had a substantial 12 effect, maybe tens or hundreds of thousands of pounds in 13 fines that Fujitsu may have had to pay. 14 Q. Do I understand from what you said a couple of answers 15 ago that you were saying that you understood your 16 primary aim was to get the system up and running and 17 working, back on the road, so that those financial 18 penalties were either not suffered or were minimised -- 19 A. Yes. 20 Q. -- rather than necessarily taking a fundamental look at 21 what the underlying or root cause was? 22 A. It was widely accepted that the underlying or root cause 23 was that the system was crap. It needed rewriting. But 24 that that was never going to happen because the money 25 was not available, the resources were not available to 22 1 do that. It was being looked at behind the scenes, and 2 a web version was being considered, from what 3 I remember. One of the problems was that the suppliers 4 of the Riposte system, from what I remember, they 5 couldn't -- it would have been very bad if we -- if 6 Fujitsu had told them that we were going to move away 7 from their product because they were still supporting us 8 and supporting it. So if they'd known the rug was going 9 to be pulled from under their feet, as it were, they may 10 not have been as co-operative as they were. 11 Q. Was it the case that sometimes, nonetheless, the errors 12 in coding were passed on to the software developers 13 within Fujitsu to fix? 14 A. Yes, if we found a definite bug then we would pass it on 15 to them to fix. We wouldn't fix the bugs ourselves. 16 Q. How would the bug be passed on to the software 17 developers to fix? 18 A. I can't remember. 19 Q. Can you remember, in terms of names, any of the software 20 developers that would have these issues passed to them? 21 I realise that we're two decades on now. 22 A. No, I have a very poor memory for names and I can't 23 remember any. 24 Q. You say in paragraph 9 here: 25 "We regularly identified issues with the computer 23 1 coding in the Horizon System. We would then flag those 2 issues to the Fujitsu IT software developers. The 3 developers would then work on a 'fix' while we monitored 4 the whole estate in relation to that issue." 5 Is that right? 6 A. Yes. 7 Q. Now, you were being asked to look at an issue on the 8 back, essentially, of a subpostmaster complaint; is that 9 right? 10 A. I was often asked to look at issues because of 11 complaints from subpostmasters, yes. 12 Q. But, presumably, if a coding error was discovered as 13 a result of the investigation of that complaint or some 14 data corruption, that could potentially have affected 15 hundreds or even thousands of other transactions with 16 other subpostmasters? 17 A. Yes. 18 Q. Was there any process to identify whether any other 19 transactions were afflicted by the bug that was 20 discovered? 21 A. I think so but I can't remember for definite. 22 Q. Can you remember whether that was an SSC task or 23 somebody else's task? 24 A. It would have been an SSC task. 25 Q. So, trying to jog your memory, if I can, a little, would 24 1 it be part of the SSC's task to put right the 2 consequences of a bug that had been discovered, not just 3 for the subpostmaster who had raised the issue but for 4 a wider range of subpostmasters? 5 A. Yes. 6 Q. Can you remember whether the other subpostmasters' data, 7 that may have been afflicted by this bug, were notified 8 of the cause of the discrepancy or error in their own 9 data? 10 A. I can't say definitely but I'm fairly sure that they 11 weren't. 12 Q. So were they told "There's an error in your data, it's 13 going to be corrected, here's the correction"? 14 A. That specifically: sometimes yes, sometimes no. 15 Q. So sometimes they weren't even told that their data was 16 being corrected; it was corrected without their 17 knowledge? 18 A. Yes. 19 Q. Sometimes they were told that their data was being 20 corrected? 21 A. Yes. 22 Q. But your memory is that they weren't told the underlying 23 reasons why it was flawed or affected in the first 24 place, ie "This is a software bug within Horizon"? 25 A. That's what I remember, yes. 25 1 Q. When you were dealing directly with a subpostmaster, say 2 the person that had raised the issue, the complaint, did 3 you explain to them that their problem had, on 4 investigation, been found to have had, as its root 5 cause, a coding error or bug within Horizon? 6 A. Quite often we'd identify the problem with the data on 7 the counter, we'd know what was wrong with that so we'd 8 be able to fix that, but we wouldn't know at that point 9 what had caused it so if we were talking to the 10 postmaster, we would have just say that we'd identified 11 the problem "with your counter, there's been data 12 corruption, or something, and we need to fix it, so we 13 need to do this, whatever, to fix the problem, otherwise 14 there will be a problem with your account". 15 Q. So it wasn't habitually fed back to them that it was 16 a coding error, or multiple coding errors, that had 17 caused the underlying problem? 18 A. No. 19 Q. Was there an official line on this as to whether or not 20 you should or shouldn't tell subpostmasters what the 21 underlying causes of these data errors or corruption 22 were? 23 A. I can't remember if there was an official line or not. 24 Q. But the practice was to not tell them? 25 A. Yes. 26 1 Q. Can we turn to paragraph 17 of your Inquiry witness 2 statement, please, which is on page 7 at the foot. You 3 say: 4 "In my opinion the coding and development of the 5 system did not meet my expectations of quality for 6 a major software project; I considered it to be a very 7 poor system that should never have been deployed but 8 I cannot be more specific than this." 9 Does that reflect the epithet that you applied more 10 pithily earlier as to your overall view of the system? 11 A. Yes. 12 Q. Can we turn back, please, to paragraph 11, which is on 13 the previous page. You say: 14 "Sometimes we were instructed not to let the 15 [subpostmaster] know we had altered his system whilst he 16 was logged on -- to my recollection, sometimes POL 17 requested this, sometimes Fujitsu, and sometimes only 18 our department knew of it." 19 A. Yeah. 20 Q. Where did the instruction come from, from within 21 Fujitsu? 22 A. I have no idea. 23 Q. Who was communicating that instruction to you? 24 A. It would have come from the manager, Mik Peach. 25 Q. If the instruction came from POL, did it come directly 27 1 from POL to you, the Post Office to you, or did it go 2 via Mik Peach? 3 A. It went via Mik Peach. 4 Q. So one way or another, instructions not to let the 5 subpostmaster know you had altered system whilst they 6 were logged on came through Mr Peach? 7 A. Yes. 8 Q. Can you remember whether there was any discussion in the 9 office at the time about whether it was important to 10 notify the subpostmaster community more broadly of the 11 finding of a Horizon System error and that this was 12 causing or could cause discrepancies of data? 13 A. I can't remember there being any discussion about that. 14 It was, as far as we knew, it was notified through Mik 15 Peach, through the development teams and through to POL. 16 If the chain of management was working correctly, then 17 POL would have been informed and then it was down to POL 18 to inform their managers that there was a problem. 19 Q. When you were speaking to subpostmasters and you said 20 sometimes you would tell them that "We've investigated 21 and we found that this is the problem". Would you ever 22 say, "Look this is an issue we've come across before. 23 Don't worry, it's not you, it's the system. We've had 24 a number of reports like this"? 25 A. We would have -- I'm sure that on occasion we said 28 1 "We've seen this before, it'll only take a few minutes 2 to fix", or something along those lines, yes. 3 Q. You mentioned earlier your view of the Horizon System. 4 Could we look, please, at POL00029991, and look at 5 page 2, please, and look at paragraph 10. This is your 6 first witness statement in the High Court proceedings, 7 Mr Roll. In paragraph 10 you say: 8 "My recollection is that the software issues we were 9 routinely encountering could, and did, cause financial 10 discrepancies at branch level, including 'shortfalls' 11 being incorrectly shown on the Horizon System." 12 Just stopping there, you say "software issues [you] 13 were routinely encountering could, and did, cause 14 financial discrepancies". Can you expand at all or 15 explain what you mean by "routinely encountered"? Was 16 it a daily occurrence or a weekly occurrence? 17 A. Um ... I would say that my recollection would have been 18 a weekly occurrence within the team. 19 Q. Was that consistently so over the three and a half years 20 that you were in the SSC? 21 A. There were times when maybe some new software had been 22 released and that would be a bit buggy, so there would 23 be times when we were having multiple issues and it was 24 very, very busy. At other times, we were able to work 25 on some -- we would have been a bit quieter so then we 29 1 would try to work on other things that had been maybe 2 put on the back burner but I couldn't really go into any 3 more depth than that. I can't really remember any of 4 the details. 5 Q. Thank you. Can I just explore, so that I -- my 6 understanding of what you are saying is completely 7 accurate. You said that after a new release, the system 8 might become a bit buggy. Do you mean there would be 9 a spike in reports of discrepancies following the 10 release of some new software? 11 A. Yes. There might be more reports from the postmasters 12 or we might find more problems with our monitoring 13 systems that we'd set up to monitor the system to make 14 sure everything was running smoothly. Sometimes the 15 postmasters would not have been aware of the problem. 16 They wouldn't have seen it, but we'd have picked it up 17 so we'd then fix it, and not necessarily by going into 18 the counters or anything, but just by manipulating the 19 data further along the line. 20 Q. Looking at the three and a half year period as a whole, 21 and putting aside the peaks and troughs that you've just 22 described, over the course of that three and a half year 23 period, did the position get any better or worse or did 24 it just stay the same? 25 A. I think it improved. As time went on, standards of 30 1 coding improved and of the documentation, but that's 2 a distant memory now and I can't really remember 3 definitely. 4 Q. What was the cause of the improvement in standards of 5 coding? 6 A. I just think people were being more professional about 7 it. 8 Q. Why were they being more professional about it? 9 A. I don't know. Maybe -- I don't know. 10 Q. Which people are you talking about? Are you talking 11 about the people in the software development arm? 12 A. Yes. 13 Q. When you joined in early 2001 and over the course of the 14 first year, did you form a view of how reliable the 15 Horizon cash accounts were? 16 A. Yes. 17 Q. What was your view? 18 A. It was pretty ropey. I said to Mik, the manager, at one 19 point that "Surely, this should be rewritten". His 20 reply was "Yes, but it's never going to happen", or 21 something like that. I think I mentioned that before. 22 Q. The "it's that never going to happen", was that for the 23 reasons that you gave earlier: money and the damage of 24 a relationship between Fujitsu and Riposte? 25 A. Money, relationship damage, also we just didn't have the 31 1 staff, which comes down to money, again, yes. 2 Q. You tell us at the end of paragraph 10: 3 "If we were unable to find the cause of the credible 4 then this was reported up the chain and it was assumed 5 that the postmaster was to blame." 6 A. That's my belief, yeah. 7 Q. Who was it assumed by? 8 A. Post Office, I believe, and the management of, probably, 9 Fujitsu. 10 Q. Do you know how such a decision or how such 11 an assumption was made by them? How they came to assume 12 it? 13 A. No. 14 Q. Do you know who was involved in reaching that view? 15 A. No. 16 Q. But the way you expressed it, makes it sounds as if it 17 was by -- a view was reached by default? 18 A. That was my feeling. If we couldn't find a problem with 19 the system, if we couldn't work out why there was an 20 error or why there was a problem, then the position, 21 from what we -- from what I understood, was that if we 22 can't find the problem in the code or in the data, there 23 is no problem. So, therefore, if there's no problem 24 with the system, it must be the postmaster. 25 Q. Did you understand that action was therefore taken 32 1 against subpostmasters? 2 A. No. At the time we would be looking at this, it could 3 be years later before any action was taken. That's my 4 understanding. 5 Q. An assumption that it must be action by or wrongdoing by 6 a subpostmaster doesn't sound like a very strong 7 foundation to take action against them, as opposed to 8 proof positive that they had done something wrong. How 9 comfortable with what was happening did you feel at the 10 time? 11 A. At the time, we didn't know any action was going to be 12 taken. 13 Q. Were you aware that people were being prosecuted? 14 A. Not at that time. 15 Q. In the first year of working, so early 2001 onwards, did 16 you hear that anyone in third line support or indeed 17 fourth line support was asked to be an expert witness in 18 a Horizon prosecution at Kingston Crown Court? I'm 19 referring to the case of Tracy Felstead? 20 A. I can't remember. I don't think so. 21 Q. If we scroll down, please. In paragraph 11, in the 22 first sentence, you tell us that there were over 30 23 individuals working on the same floor -- I've asked you 24 about that already -- and that your recollection was 25 that many of those individuals were involved in similar 33 1 work or other Horizon related IT work. Then in the last 2 sentence, you say this: 3 "I would describe much of the work being carried out 4 as 'firefighting' coding problems in the Horizon 5 System." 6 I just want to understand what you mean by that. 7 I understand "firefighting" to mean spending time on 8 problems that need to be dealt with quickly instead of 9 working in a calm, ordered and planned way. Is that the 10 sense in which you meant it? 11 A. Yes. 12 Q. What was it like working in such an environment? 13 A. It was quite hectic at times. Sometimes there'd be 14 a bit of a panic on and it would be all hands on deck to 15 get a -- fix a system as quickly as possible. That's 16 all I can say, really. 17 Q. Thank you. Can we look, please, at the second witness 18 statement you provided in the High Court proceedings, 19 and that's POL00042225. Can you see this is your second 20 witness statement, dated 16 January 2019. 21 A. Yes. 22 Q. Can we go to the fourth page, please, and look at 23 paragraph 12. Here I think you've been asked to reply 24 to or comment on certain paragraphs in a report produced 25 by the defendant, Post Office, Dr Robert Worden, and you 34 1 say in paragraph 12: 2 "At paragraph 167 Dr Worden describes software 3 errors being corrected by Transaction Corrections, and 4 [he] states 'If there were any such software error, it 5 would probably occur with such high frequency, and occur 6 uniformly across all branches, giving rise to so many 7 [Transaction Corrections], that Post Office would soon 8 suspect a software error (for instance, seeing the 9 effect repeatedly in some MIS report) and require 10 Fujitsu to correct it'." 11 You say: 12 "I do not recall Fujitsu carrying out any analysis 13 of Transaction Corrections to try to identify if there 14 may be an underlying software error. I also think it is 15 wrong to say that software errors would occur uniformly 16 across branches as [you] explained ... above. My 17 experience was that software errors occurred in very 18 specific factual circumstances, which is why they were 19 challenging to identify and correct." 20 Is what you say there accurate? 21 A. Yes, I believe so. 22 Q. This tends to suggest that, in your team, there wasn't 23 any underlying analysis -- or, sorry, any analysis of 24 underlying root causes; would that be fair? 25 A. I'm not sure I can really remember now. If we were 35 1 getting lots of calls in, then -- for a specific or very 2 similar problem, you know, within a period of a couple 3 of days, then, you know, you'd be very aware of that 4 and, if that was the case, then sometimes we would have 5 been probably aware of that and worked on a fix before 6 POL were even aware of it. 7 Q. I'm more getting to the issue of whether the Post Office 8 came to you and said "We suspect a software error. Can 9 you conduct", I don't know "some meta analysis of the 10 system to see whether our suspicion is correct"? 11 A. I don't think the Post Office ever came to us to say 12 that. I can't remember for sure but I'm pretty certain 13 they didn't. 14 Q. Thank you. That can come down now. 15 Were you aware of a team called the Customer Service 16 Security Team? 17 A. I don't remember that phraseology, no. 18 Q. Can you recall or remember somebody called Andrew or 19 Andy Dunks? 20 A. No. 21 Q. Can you recall a job title or role being undertaken of 22 the cryptographic key manager? 23 A. There was a key, which was a crypto key, if you like, 24 which was generated by a secure PC in a locked room 25 within the SSC, bearing in mind that the SSC itself was 36 1 on the sixth floor of a very secure building behind 2 double doors that were extremely secure. It was a very, 3 very secure area. But that's about all I can remember. 4 Q. Mr Dunks was the manager of the cryptographic key. 5 We've heard from him recently. I think it follows from 6 what you've said that you didn't have any or you don't 7 recall any liaison with him or the security team? 8 A. No. 9 Q. We know that he, the cryptographic key manager, was 10 selected to give evidence by provision of witness 11 statements and giving oral evidence in court, about what 12 you and your team in the SSC had done in response to 13 calls to the SSC and the work that your team had 14 undertaken as recorded on call logs. Do you understand? 15 A. Right. 16 Q. Do you know why one of that team, the customer service 17 team, and, in particular, the person that managed the 18 cryptographic key, was selected to give evidence about 19 what you and your team were doing in the SSC? 20 A. No. 21 Q. Were you ever party to a discussion or did you ever hear 22 about why somebody who managed the cryptographic key 23 would give evidence about what some other people were 24 doing, rather than you or somebody in your team giving 25 evidence? 37 1 A. Sorry, can you repeat the question? 2 Q. Yes. Did you ever hear any discussion or were you ever 3 party to any discussion about why Mr Dunks, the crypto 4 key manager, was giving evidence about what was or 5 wasn't shown on helpdesk call logs that were completed 6 by you and members of your team, rather than a member of 7 you and your team giving evidence? 8 A. No. 9 Q. Did anyone ever ask you to give evidence about what you 10 did in response to any calls or raising of concern about 11 data errors or discrepancies? 12 A. I don't think so. 13 Q. If they had have done so, would you have described all 14 of the issues and the problems that we are discussing 15 here today? 16 A. Probably, yes. 17 Q. Did you ever hear any discussion about who from Fujitsu 18 should attend court to give evidence about the operation 19 of the Horizon System? 20 A. I don't recall ever hearing anything about that, no. 21 Q. In your time, did you know whether anyone from Fujitsu 22 was to attend or had attended court giving evidence 23 about the operation of the Horizon System? 24 A. I can't remember that happening. 25 Q. Thank you. Can I turn to some hardware issues, please. 38 1 Can we have up, please, POL00029991. This is your first 2 witness statement, in the High Court proceedings again, 3 and if we turn to the third page, please, and look at 4 paragraph 14 at the top, you say: 5 "As well as software issues, I can also recall that 6 there were regular IT hardware issues at branch level. 7 However, I would reiterate that the main recurring 8 issues were software issues." 9 Could hardware issues affect the integrity of the 10 data recorded or produced by Horizon? 11 A. Yes. 12 Q. What hardware issues would typically affect the 13 integrity of the data recorded or produced by Horizon? 14 A. If the database on one of the counters became corrupted 15 then it could stop that counter communicating with the 16 rest of the system, which would lead then to 17 transactions being marooned on that counter. Depending 18 on what the problem on the counter was, it may have been 19 a fairly quick fix, maybe we could just fix it on the 20 counter itself, or it may have been that we had to get 21 the counter back into Bracknell where one of the guys 22 would hack into it and retrieve the data. 23 Sometimes, if the counter was beyond recovery, then 24 transactions could be lost, so bills that had been paid 25 may not have gone through or whatever money that had 39 1 been paid to the counters -- to the post office, might 2 not have been recorded properly. 3 Q. Thank you. That can come down. Can we look, please, at 4 POL00042225. This is your second witness statement. 5 Can we go to page 2, please, and look at paragraph 5 6 under the heading "Hardware Failures". You say, 7 "Dr Wardon refers", and you remember what you were doing 8 in this statement: 9 "Dr Worden refers at paragraph 151 of his report to 10 hardware failures. He says 'Although the hardware in 11 the branches was not always reliable and communications 12 infrastructure at that time were not highly liability, 13 there were strong measures built into Old Horizon to 14 ensure that hardware failures and communication failures 15 could not adversely affect the branch accounts'." 16 You say: 17 "During my time at Fujitsu we frequently encountered 18 hardware failures which had occurred in branches and 19 required our intervention to attempt to remedy the 20 problem. I would estimate that I was involved with 21 a hardware failure on average at least once a month. 22 These problems could and did affect branch accounts." 23 Is that correct? 24 A. Yes. 25 Q. At paragraph 6, you say: 40 1 "The most extreme case that I can recall was 2 a complete failure of a counter to communicate with the 3 server, which required the counter to be removed to the 4 SSC so that the data could be recovered, and 5 a replacement counter installed in the sub post office. 6 Prior to the problem being identified, data could be 7 backing up on the counter without it being replicated to 8 the other counters or to the correspondence server." 9 Is that correct? 10 A. Yes. 11 Q. Is that what you were alluding to a moment ago? 12 A. Yes, I can definitely remember one where we had it -- 13 more than one where they were brought back for the data 14 to be recovered and then put back into the system later. 15 I can't for 100 per cent recall whether we had one where 16 we couldn't recover all the transactions but I'm fairly 17 sure we did have one where we didn't -- 18 Q. I'm sorry, I missed what you said at the end there? 19 A. I'm fairly sure there was one or more occasions where we 20 couldn't recover all the data but I can't say that for 21 certain. 22 Q. Can we skip to paragraph 8, please. You say: 23 "I recall one particular case where branch data was 24 not being replicated from a mobile post office correctly 25 and it appeared that the subpostmistress was turning off 41 1 the power mid transaction. As we could not fix this 2 problem over the phone with the subpostmistress, she 3 sent the laptop to Fujitsu for examination. Using the 4 Post Office test rigs on the sixth floor, and comparing 5 the results with the laptop that had been returned to 6 Fujitsu, I discovered that the button which should have 7 put the laptop into standby mode was actually switching 8 off the power, resulting in the disk crashing. 9 I disassembled the laptops to confirm this. Thus, when 10 the postmistress thought she was switching her counter 11 to standby mode, which would have initiated a controlled 12 shutdown and allowed the datastore to replicate the 13 servers, she was actually switching the power off, which 14 is what we were seeing in the SSC. When I raised this 15 with my manager, Mik Peach, who subsequently talked to 16 the hardware team, I found out that this was a known 17 problem: one of the engineers had made a mistake with 18 a batch of laptops which had been sent out to branches 19 before the error was detected. No one outside the team 20 responsible for building the laptops had been informed 21 of this meant that I had spent several days 22 investigating the problem. Whereas the subpostmistress 23 in this case was provided with a replacement laptop, 24 knowledge of this problem was kept within the 25 departments concerned and the batch of faulty laptops 42 1 was not recalled. It is my belief that Fujitsu senior 2 management and Post Office were not informed." 3 Is that all correct? 4 A. Yeah. 5 Q. When you're referring to Fujitsu senior management not 6 being informed, who were you referring to, what level? 7 A. Well, my manager knew, Mik Peach, his friend who ran the 8 build team knew. Whether Mik ever told his manager, 9 I don't know. As far as I'm aware, it never got up the 10 chain beyond that. I was told to basically hush it up. 11 Q. Why were you -- what words were used to tell you 12 basically to hush it up? 13 A. I can't remember exactly but it was -- it had been dealt 14 with. 15 Q. Who told you basically to hush it up? 16 A. Mik. 17 Q. In an answer a couple of answers ago, you say you don't 18 know whether it went any further. Here, you say it's 19 your belief that it didn't go any further, that Fujitsu 20 senior management were not informed. What was that 21 belief based on? 22 A. The way I was asked to close the call and the fact 23 that -- I can't remember exactly it's just that -- the 24 way I was told to deal with the caller and to get rid of 25 it. 43 1 Q. Was that the only time that that kind of thing was said 2 to you? Was this an isolated example, so "Keep it 3 within the team", or did that happen on more than one 4 occasion? 5 A. That is the only one that really sticks in my memory. 6 I can't remember if it happened on more than one 7 occasion. 8 MR BEER: Thank you. 9 Sir, it's quarter past now. I wonder whether that 10 might be an appropriate time for the morning break. 11 SIR WYN WILLIAMS: Yes, certainly. 11.30 all right, 12 Mr Beer? 13 MR BEER: Yes, thank you very much. 14 SIR WYN WILLIAMS: All right, see you again at 11.30, 15 Mr Roll. 16 THE WITNESS: Thank you. 17 (11.13 am) 18 (A short break) 19 (11.30 am) 20 MR BEER: Sir, good morning, can you see and hear me? 21 SIR WYN WILLIAMS: Yes, I can. Thank you. 22 MR BEER: Can you, Mr Roll? 23 A. Yes, thank you. 24 Q. Thank you very much. Can we turn up a document, please, 25 with the URN FUJ00086267. You'll see, from the bottom 44 1 right of the document, that this is dated 2011, so 2 post-dated by many years at the time of your leaving 3 Fujitsu. If you scroll to the top, please, you will see 4 under the abstract that it concerns HNG-X, Horizon 5 Online, of which you were not a part, correct? 6 A. Correct. 7 Q. But I want to ask you about whether something within the 8 document replicates the position when you were working 9 for that three and a half year period for Fujitsu. Can 10 we turn, please, to page 15 and look down to 11 paragraph 2.7, "Removal of duplication". If we just 12 read it together: 13 "All support groups should ensure that they do not 14 pass to the right duplicate incidents, ie incidents 15 which are repetitions of an incident which has already 16 been passed to the next line of support. They should 17 either retain the duplicate incidents within their own 18 call logging system or close them as duplicates: 19 "1st line units retain duplicates under a 'master 20 call' and to ensure that when the resolved incident is 21 received from 2nd line, the end user is contacted and 22 duplicated call incidents closed within TfS. 23 "2nd-4th line support units normally immediately 24 close the incidents as duplicates because they add no 25 value to the support process at these levels. This 45 1 results in the incidents being returned to 1st line ... 2 "Duplicate incidents are only acceptable where the 3 symptoms reported by the customer did not match the 4 symptoms recorded in the original incident, and which 5 therefore could not reasonably have been identified as 6 a duplicate. 7 "Failures will be reflected in filtration figures 8 where the incidents are closed in the 'duplicate 9 incident' category in PEAK by subsequent support units." 10 Does that reflect the working practice of the SSC at 11 the time you were in post? 12 A. I can't remember. 13 Q. Can you remember any instructions on the treatment of 14 duplicate incidents? 15 A. No. 16 Q. Can you remember any instruction, custom or practice, 17 the effect of which was to minimise or seek to minimise 18 the reporting of duplicate incidents, and that they 19 would be regarded as a black mark against the support 20 team concerned? 21 A. Not sure. I think I -- they may have been returned to 22 first line support because we were already looking at it 23 but I can't remember for sure. 24 Q. Okay, I understand. Do you remember Anne Chambers? 25 A. I remember the name but I wouldn't recognise her. 46 1 I couldn't -- I don't know her. I only remember the 2 name because the name has come up recently. 3 Q. Do you remember that person, even though you wouldn't 4 recognise her, as a person who worked, in your time, at 5 the SSC? 6 A. Yes. 7 Q. Can you recall whether she had any particular expertise? 8 A. I think she was very good on the accounting side, as she 9 was, I think, very experienced in going through the 10 databases but I can't remember, really. 11 Q. Did she, to your knowledge, have any expertise in the 12 integrity of the software on Horizon -- 13 A. I can't remember. 14 Q. -- or on the integrity of Horizon data? 15 A. I can't remember. 16 Q. In your time, noting the time at which you left, did you 17 have any conversations with her about a requirement for 18 her to give evidence in any court proceedings? 19 A. No, not that I remember. 20 Q. In your time, can you recall whether she was selected to 21 give evidence in any court proceedings? 22 A. No, I don't remember. I don't recall anybody being 23 selected but, from what you've said, they were, but 24 I have no recollection of it. 25 Q. Can we look, please, at POL00073280. This is an exhibit 47 1 sheet, so it's like the front sheet of an exhibit 2 produced by Mr Dunks, Andrew Dunks, who I mentioned 3 earlier, and in it is a selection of call logs produced 4 by Mr Dunks for the purposes of some civil proceedings 5 that the Post Office took against a man called Lee 6 Castleton. 7 Can we turn to one of those call logs, please. It 8 starts on page 20. Just if we can expand it out so you 9 can look at the whole of the first page of it. Do you 10 recognise the format of this call log? 11 A. No. 12 Q. At the time, did you ever look at call logs in printed 13 format or would they appear on the screen to you? 14 A. I think they were always on the screen. 15 Q. You'll see, and bearing in mind that you wouldn't have 16 seen it in this format, if we look at the top we can see 17 that the call was opened on 25 February 2004. Can you 18 see that -- 19 A. Yeah. 20 Q. -- in the middle at the top? So that's within your time 21 working on the SSC. 22 A. Yeah. 23 Q. Can you see in about ten boxes below under "Problem 24 Text" it says "pm", which I think is postmaster: 25 "[Postmaster] reporting that they are getting large 48 1 discrepancies for the last few weeks." 2 A. Yes. 3 Q. Yes? Just so you know, this call relates to 4 difficulties that Lee Castleton was having at his 5 branch. I just want to run through this call log, 6 please, to see whether you can help us with what some of 7 the text means and what was done in relation to it. 8 If we scroll down, please, to "Call Activity Log", 9 which is right at the foot of the page at the moment. 10 Again, you wouldn't have seen these entries in this way; 11 they would be on a screen, is that right, for you, and 12 not set out in this format? 13 A. I can't remember. I don't know if we'd have seen any -- 14 much of -- all of this data or not. I can't remember. 15 Q. Let's just go through it and see whether looking at it 16 in a bit more detail and slowly helps you. Do you see 17 the first entry "OPEN": 18 "New call taken by Kuljinder Bhachu ..." 19 This is on 25 February 2004: 20 "... [postmaster] reporting that they are getting 21 large discrepancies for the last few weeks." 22 That's what we read above. 23 Is that the way the SSC would operate, by putting 24 a pithy summary of the text within an entry like that? 25 A. I can't remember. 49 1 Q. Okay, moving to the next line, also on 25 February: 2 "Looking at closed calls for this site, there have 3 been a number of calls logged regarding discrepancies. 4 NBSC have been in contact with the [postmaster] and 5 cannot find any user error." 6 Can you now remember what NBSC was? 7 A. No. 8 Q. Okay. The next line, also later that day: 9 "Spoke to Sandra [and] NBSC ... regarding this 10 issue. Checked Tivoli events and health checked. Site 11 is health checking ok." 12 Can you now remember what Tivoli was? 13 A. I think that was a software program that ran in the 14 background and monitored events and set alerts if it 15 detected anything, any errors. 16 Q. Next entry: 17 "Critical event scene @ [and a time is given on 18 18 February] stating 'Error message. An error has 19 occurred = see the audit log'." 20 The next entry later still that day, "KEL 21 Reference". 22 Can you remember now what KELs were? 23 A. Yeah, that was the Known Error Log. That's about all 24 I remember of it. 25 Q. Can you remember what the Known Error Log was? 50 1 A. A log of known errors. 2 Q. Who was it maintained by? 3 A. I can't remember. 4 Q. Was it maintained by the SSC? 5 A. I can't remember. 6 Q. Next entry, later still: 7 "Downloading event logs for progression [and some 8 numbers] application ... system & ... security." 9 Next entry, a little later still: 10 "Previous history in calls ..." 11 Then some references are given. 12 Next entry: 13 "Spoke to [postmaster], who advises that the problem 14 with the CA ..." 15 Do you remember what "CA" was? 16 A. I think it's cash account. 17 Q. "... started ever since the BT engineer came to move the 18 BT box for the preparation for the installation of 19 ADSL." 20 A. Yeah. 21 Q. Next entry, later still: 22 "[Usernames are given] Other BAL users ... stock 23 unit aa balance on Wednesday after 17.30." 24 Does this mean anything to you so far? 25 A. Not really. You've got two -- CTR001 is just 51 1 a username. So that's all I can say from that. 2 Q. Then this: 3 "Could SSC please investigate why this [post office] 4 is experiencing large discrepancies ever since BT 5 engineer has moved BT box in preparation for ADSL 6 [installation]. KEL [reference] given as possible 7 problem. NBSC have said there is no user error." 8 Would you understand that last entry to mean that 9 "It's not the subpostmaster that's doing anything 10 wrong", say NBSC? 11 A. Yes. 12 Q. Then skipping to the foot of the page, bar one entry, 13 an entry by Barbara Longley: 14 "Incident Under Investigation Prescan: Assigning 15 call to Anne Chambers in EDSC." 16 Can you recall what EDSC was? 17 A. No. 18 Q. Can we go over the page, please. An entry by Anne 19 Chambers on the 26th: 20 "Incident Under Investigation. KEL quoted is 21 relevant -- if the audit log had been checked, it would 22 have shown a different error message. The event was 23 part of a storm which occurred over the estate that 24 night as a result of a faulty software fix, and has 25 nothing to do with the discrepancies." 52 1 Can you help us with what "a storm occurring over 2 the estate" might refer to? 3 A. I think it refers to a whole load of errors that were 4 generated but, I must admit, I'm guessing there. 5 I can't remember. 6 Q. Okay. The next entry: 7 "No transaction date and time was provided for this 8 transaction using current date and time." 9 Then an entry by Anne Chambers: 10 "Advice and guidance given. I have checked various 11 things on the system. All the internal reconciliation 12 checks are okay. Cheques are being handled correctly 13 (except for 10th Feb when the clerk forgot to cut off 14 the report -- but this didn't cause a discrepancy). 15 Cash declarations look okay, they usually use drawer 16 ID11. Occasionally they have used a different drawer 17 ID, this can lead to amounts apparently doubling on the 18 cash flow report, and should be avoided. But again it 19 will not cause a discrepancy. Checking the cash 20 transactions on the system against the declarations 21 shows that they're not working particularly accurately, 22 (ie at the end of the day the cash they declare in the 23 drawer is tens, hundreds or thousands of pounds astray 24 from what has been recorded on the system). It is 25 possible that they are not accurately recording all 53 1 transactions on the system. There is no evidence 2 whatsoever of any system problem. I've mentioned this 3 outlet to Julie Welsh (Customer Services) who will try 4 to get POL to follow it up, but in the meantime please 5 tell the [postmaster] we have investigated and the 6 discrepancies are caused by the difference between the 7 transactions they have recorded on the system and the 8 cash they have declared, and are not being caused by the 9 software or hardware." 10 Then there's some entries that don't concern us. 11 Can you tell what Anne Chambers has done, from these 12 records, in order to reach these conclusions? 13 A. No. 14 Q. What would, typically, you do when presented with the 15 problem that Anne Chambers was presented with? What 16 investigative steps would you undertake? 17 A. There's not a lot of information in the call log for me 18 to give you much of an answer to that. I don't think 19 there was any specific figures given. I can't remember 20 what I'd have done in this situation. 21 Q. Can you remember a species of data called ARQ data? 22 A. No. 23 Q. Can you help us, and given the answers to the questions 24 I've asked so far, I think it might be limited, the 25 help, you can give us, how a KEL would be used to 54 1 investigate a call like this? 2 A. I'm trying to remember. The KEL would have information 3 about what the symptoms of the problem were. It gives 4 you pointers as to what was causing the problem, so that 5 then you could go into the system and look for those 6 particular traits, if you like, to confirm that that was 7 the problem, and it would then give you the details of 8 the fix, which you could then apply to rectify the 9 problem. 10 So if they provided a KEL there but then, when you 11 looked at it, all the audit log data or whatever, event 12 logs, et cetera, didn't have the relevant information in 13 or different information in them, then that KEL wouldn't 14 apply. So that would then not be the KEL that was 15 relevant. In that case, you're then sort of working 16 blind and you've got to try to work out from what the 17 postmaster is saying where there is a problem. 18 So you'd be working on that, going through the 19 systems, the accounts, et cetera, and trying to find 20 out, if there was a problem with the counters or with 21 the software, where it was. Working blind, largely, and 22 then -- that's all I can really say. You'd have three 23 days to find the fault and then you'd have to hand it 24 back. 25 Q. We can see that, here, the helpdesk put up a KEL number, 55 1 and Anne Chambers looked at that KEL and found that it 2 wasn't relevant. Was there a way of searching the KELs 3 to look for a fault or problem that was similar to the 4 one that you were being asked to investigate? Because, 5 in this case, she's ruled out that KEL as being 6 applicable. Was there a way of, I'm imagining a keyword 7 search, or free text search, or way into the KELs, to 8 look at whether the problem that you were being asked to 9 look at was indeed a known error? 10 A. I can't remember. 11 Q. Okay, thank you. Can that come down now, please. Can 12 we look at an Excel spreadsheet document. It's 13 POL00028922. Thank you. We're looking at tab 5, and 14 it's called "Finals Count". The heading of this is 15 "Total PEAKs resolved" by you, between 21 March 2011 and 16 17 September 2004. That roughly accords with the period 17 of time that you spent in the SSC, doesn't it? 18 A. Approximately, yes. 19 Q. Do you know the provenance of a document like this? 20 A. No, no. I imagine that Fujitsu have provided it and it 21 will show some of the work that I did while I was there. 22 Q. It appears to be a record of result codes and a total of 23 them, on the right-hand side, attributed to you. I just 24 want to ask for your help, please, in whether you can 25 remember what any of the result codes are or, more 56 1 particularly, the kind of problems and the resolutions 2 of them that might occur. Do you see the first one is 3 "Ref Data Fix Released to Call Logger"? 4 A. That's a reference data fix. Sometimes the reference 5 data was corrupt or incorrect and so we'd have to send 6 out a fix. The reference data, that's the reference 7 data being all the information regarding, for instance, 8 stamps, or fishing licences, or gas companies, utility 9 companies, that sort of thing. 10 Q. So the cost of items supplied by third-party suppliers 11 that the Post Office administered, essentially? 12 A. Costs, yeah, or maybe bank account -- no, that's 13 probably a bad one. But address details or -- yeah, 14 just -- not just costs but product details fully, you 15 know, everything to do with the product. 16 Q. Thank you. A reference data fix, what would that 17 involve? 18 A. I can't remember. I made some -- one of the teams would 19 have to rewrite the database that held all the data and 20 then redistribute it to the estate or to the relevant 21 post offices. 22 Q. Why might the reference data be wrong or require fixing? 23 A. Somebody had keyed it in wrong. 24 Q. The next one, underneath, "S/W Fix Released to Call 25 Logger". 57 1 A. That's software fix. 2 Q. What would that refer to, which software and where? 3 A. I'm not sure. I mean, there were so many areas of 4 software, not just the Riposte system that the counters 5 were running in there. I can't remember the full 6 details. 7 Q. But software within the Horizon System? 8 A. Somewhere within the system, yes. 9 Q. The third of them "Build Fix Released to Call Logger"; 10 what would a build fix release be? 11 A. I think that relates to the NT software that was running 12 on the counters. So you had the basic counter, which 13 was -- it had NT installed on it but it was very -- 14 that's Windows NT. It was a very doctored system, so 15 that then the Riposte system sat on top of the NT system 16 and on top of Riposte, from what I remember, there was 17 the Horizon System. So the build fix, I think, referred 18 to the NT, which was the basic box. If there'd been 19 a software upgrade to the Windows software that maybe 20 hadn't got through to that particular counter, that 21 could then cause a problem later when newer software, 22 newer Horizon software was downloaded. If that relied 23 upon Windows being up to date but Windows wasn't up to 24 date in that counter, that could have caused a problem. 25 Q. Thank you. "No fault in product". That may appear 58 1 self-explanatory and at the risk of getting that kind of 2 response from you again, can I just check what that 3 does, in fact, refer to? 4 A. It means basically that, in the time we were allowed, we 5 couldn't find a problem. 6 Q. A "product" is what, in that sentence? 7 A. Anything within the Horizon System. So it could be at 8 the backend, where it's processing overnight; it could 9 be on the counters. As I say, it doesn't mean there 10 wasn't a fault; it just meant that we couldn't find it. 11 Q. You said "in the time that we were allowed". Was there 12 a hard deadline on the amount of time that you were 13 permitted to devote to investigation? 14 A. From my recollection, we were allowed three days. 15 Q. The next one "Published Known Error". Can I ask, who 16 would the "Published" refer to: "published" to whom? 17 A. That was -- from what I remember, it was an error that 18 had been confirmed and it had been -- the details had 19 been promulgated to the first and second line with a fix 20 or within an explanation or whatever, so that it should 21 never have been sent to third line investigation because 22 it had already been investigated and the problem was 23 found. So it should have been dealt with at first or 24 second line. 25 Q. Then an "Unpublished Known Error". Why might some known 59 1 errors be unpublished? 2 A. I can't remember. 3 Q. Can you try and think back? 4 A. I can't remember. 5 Q. "Solicited Known Error"; to what did that refer? 6 A. I can't remember what that was. 7 Q. "Administrative Response", which seems to be one of the 8 higher numbers. What was an "administrative response"? 9 A. That was a general catch-all. If you couldn't work out 10 which one it should go in, then sometimes you just chuck 11 it down as an administrative response. That's what 12 I think it was. 13 Q. When you say "chuck it down", you would apply a result 14 code -- 15 A. Yeah, you had -- 16 Q. -- of, in this case, 70, I think it is, to that? 17 A. Yeah, I think that's what it was. There were certain 18 areas where it was -- it wasn't clear which one you 19 should put it in. So that was, yeah, just -- I think 20 that was the sort of catch-all. 21 Q. "Avoidance Action Supplied". Arising from that -- and 22 it's a two-parter -- firstly, what is avoidance action 23 and, secondly, to whom would it be supplied? 24 A. It would be applied to the estate so that could be to 25 the servers, but this is -- I'm not 100 per cent certain 60 1 about these, any of these, so this is what I seem to 2 remember. So from what I recall, this could be applied 3 to the servers overnight, so if the servers fell over in 4 the processing. 5 The way, when I was there, this worked, was that at 6 about 6.00 every evening, all the counters would start 7 uploading their data to the main servers, wherever they 8 were. They would be given a few hours to transfer all 9 the data and then it would all be batch processed. So 10 there were Unix programs and batches, batch files that 11 were run so they would sort the data into, you know, 12 American Express transactions and Barclays Bank 13 transactions, and all this sort of thing. 14 Then 20 minutes -- that would be given 20 minutes to 15 run, then there would be maybe another half an hour or 16 an hour, where it would add up all the figures for 17 American Express, and it would do the same for Barclays, 18 et cetera, and then another process would then run and 19 it would farm or send all the data out to another 20 database, but the next night -- because this would take 21 a long time -- processes would run to further refine 22 this data, before it was transmitted out actually to the 23 banks and to the American Express systems, et cetera. 24 So on the servers, if one of those processes fell 25 over, if you could get in quickly enough and restart it 61 1 then it would carry on running that night. Otherwise, 2 if you missed the window, you had to rerun it the next 3 night, which would then cause a bit of a backlog. But 4 if you were able to do that, that would then be 5 avoidance action because you'd got it started again and 6 avoided any sort of action. 7 If it was on the counter, it could be that there had 8 been a database corruption and you had to go in, extract 9 the data, fix the corruption, put the data back onto the 10 platform so that then the system could carry on running 11 correctly. Again, that would be avoidance action. 12 Q. Thank you. "Duplicate Call"; is that self-explanatory? 13 A. Yes. Yeah. 14 Q. That means what, a call from two different 15 subpostmasters or the same call twice -- from the same 16 subpostmaster twice? 17 A. I think it could be either. I'm not 100 per cent 18 certain now. 19 Q. "Fixed at Future release": to what does that refer? 20 A. I think that was when there'd been a problem on the 21 counter, the postmaster had phoned it in, we'd 22 investigated, found it was a known problem and that 23 there was a fix that had been written but, because of 24 the amount of data traffic on the lines, we didn't have 25 time to actually -- there hadn't been time yet to put 62 1 that fix onto the counters. So it was all ready to go 2 but it just hadn't been released yet. 3 Q. "Reconciliation -- resolved": to what does that refer? 4 A. I can't remember. Something to do with the accounting 5 but I can't remember exactly. 6 Q. "Suspected hardware fault"; that is self-explanatory. 7 A. Yeah. 8 Q. "Advice and guidance given": what kind of advice and 9 guidance might be given so as to result in this result 10 code? 11 A. Maybe it was a training issue or the postmaster was 12 doing something in the wrong order so that the figures 13 weren't adding up properly. In the previous examples 14 with that KEL, you mentioned that there was a stock 15 code -- sometimes the postmaster was using the wrong 16 drawer and that was causing issues. So that would be 17 the sort of advice that was given, you know, "Don't do 18 this because it will cause a problem". 19 Other things would be that, you know, "Don't turn 20 the computer off before 6.00 because, if you do that, it 21 may not transmit all the data", all that sort of thing. 22 Q. "Insufficient evidence": insufficient evidence to do 23 what? 24 A. To actually find out what the -- to even know where to 25 start looking for a problem. 63 1 Q. "User error": "user", does that refer to the 2 subpostmaster or counter clerk? 3 A. Either, yes. 4 Q. "Route ... to CFM"; can you remember what that was? 5 A. I can't remember what that is. 6 Q. You'll see that the total that's attributed on this 7 spreadsheet to the PEAKs resolved by you in that 8 three-and-a-half-year period was 915, so 275/300 a year. 9 Does that accord with your recollection of the work that 10 you would have got through? 11 A. I can't really remember. Quite often you'd work on 12 other -- it's not a terribly accurate way of doing 13 things, unfortunately. Sometimes three or four of you 14 would be working on a call but any one would actually be 15 recorded on it. Other times, you might be allocated 16 a call, you might be working on three or four at the 17 time, so maybe you'd pass one or two on to somebody else 18 so then they would be given as the person who'd closed 19 it. 20 Q. So you're warning us not to take too much from this. 21 All this is a record of is where you entered the result 22 code? 23 A. Yes. 24 Q. Thank you very much. That can come down now. 25 Can we turn to the issue of remote access, please, 64 1 and can we start, please, by looking at paragraph 9 of 2 your Inquiry witness statement. WITN00780100. It's 3 page 5. Just scroll down so we get paragraph 9, please. 4 Thank you. 5 Starting from the third line of your Inquiry witness 6 statement, you say: 7 "Apart from responding to requests for assistance 8 from second line, for example, looking into issues 9 reported by [subpostmasters] regarding accounting, 10 product errors, hardware failures, etc, or queries from 11 utility companies regarding payments made at [post 12 offices] that hadn't gone through, we also monitored the 13 system and ran remote programs we had developed which 14 provided advance warning of any failures, for example 15 with the overnight batch processing of network banking 16 transactions or benefits payments. 17 Then this: 18 "This sometimes meant we sometimes had to connect 19 remotely to the [subpostmasters'] Horizon terminals, 20 sometimes without their knowledge or consent, to make 21 changes to the counter configuration or the database 22 system." 23 Can I just check, Mr Roll, please, by that last 24 sentence there, are you suggesting that the changes 25 would result or could result in an alteration to branch 65 1 data that could affect branch accounts? 2 A. Yes. 3 Q. Why would, if you can remember, if you can help us, 4 making a change to the counter configuration do that? 5 A. The main one I remember is that, if the database had 6 become corrupt, if one of the transactions hadn't been 7 recorded correctly, then, although the postmaster would 8 continue to work and everything on the post office side 9 of things, on the counter would seem to be working 10 correctly, in effect, the system would be writing data 11 into the database but none of that data would then be 12 copied across to other counters or up to the servers 13 where it would be processed. 14 So, from that side of things, there could be 15 a discrepancy because the postmaster had been working on 16 the counter and yet the systems further up the line 17 wouldn't know he had done any work on it because the 18 correction would have prevented that data from being 19 read. We could then go in, into the counter, and 20 basically just correct it so that things would work 21 properly and then the data then would be harvested. 22 However, to do that, we had to take all the data off 23 the counter from the point of the corruption, save it 24 all, correct the line of code which had been corrupted 25 and then put all of the data we'd taken off back in. 66 1 If, during the correction of that line of code, we'd 2 got something wrong, we could have potentially caused 3 a problem, or, if, whilst we'd been removing the data 4 and then putting it back in, the data that the 5 postmaster had continued to enter, if we'd made 6 a mistake with that or accidentally deleted a line or 7 anything, then, again, there could have been a problem 8 there. So the other problem that could have happened is 9 that, if the postmaster hadn't been aware that we were 10 doing it and had continued to use the system or 11 accidentally use the system, then we would have 12 overwritten his data, which then would have caused 13 problems with the cash balancing and whatever. He may 14 have had more money or less money in the till than the 15 system was showing because we'd effectively deleted his 16 transactions. 17 Q. Thank you. Can we just look, please, at POL00004074. 18 Thank you. This is a transcript of the evidence you 19 gave in the High Court proceedings. I'm afraid, 20 Mr Roll, this is going to be a bit fiddly so please bear 21 with me because I'm going to be asking you about some of 22 the answers that you gave previously, all right? 23 Can we look, please, at page 34 of this document and 24 look at the bottom left-hand quadrant, which has got the 25 internal pagination 130. Can we pick it up from line 67 1 21, please. Here you're being asked questions by the 2 Post Office's barrister, or one of them, and he says, 3 quoting from your witness statement: 4 "'Still on the subject of remote access to branch 5 systems, as I recall some errors were corrected remotely 6 without the subpostmaster being aware'." 7 He says: 8 "Those errors are not errors -- or rather those 9 corrections were not corrections which changed branch 10 ..." 11 Then if we go to the top of the next page, the 12 sentence was: 13 "... which changed branch accounts in the way we 14 discussed?" 15 You answered: "No. 16 "You're talking about other errors, aren't you? 17 "Yes." 18 Question: "Could you give some examples of the kind 19 of errors you are talking about?" 20 Answer: "I can't remember, I'm afraid." 21 Then he says: "But it would be things like changing 22 configuration items?" 23 You said: "Probably, yes." 24 He said: "That sort of thing, which would not have 25 an impact on the branch accounts in the way that we have 68 1 previously discussed?" 2 You said: "I think so, yes." 3 That exchange there, and it may be difficult to 4 piece together the effect of your evidence from the 5 question and answers, but were you saying there that 6 changes to counter configuration would not have 7 an impact on branch accounts? 8 A. I can't remember exactly now. I wouldn't, I couldn't 9 definitely say that the change in the configuration 10 would or wouldn't have an effect. I just can't remember 11 that much information. 12 Q. That's very fair. Thank you very much. 13 Can we look, then, to the different routes that 14 might be taken to remote access and can we have back up 15 the fifth page of your Inquiry witness statement. 16 Page 5, at the foot of the page, paragraph 10. If you 17 just look, you say: 18 "I think there are several ways to connect to the 19 counters but it was a long time ago and I can't remember 20 the exact details. As I recall ..." 21 Then you say (a), and then if we go over the page 22 there's a (b) and a (c). So there's three ways that you 23 recall, it being a long time ago and without you 24 remembering exact details, ways to connect to the 25 counters. 69 1 I'm just going to go through each of those three 2 ways, if you don't mind. 3 A. Yeah. 4 Q. The first way, (a), if we just go back, please. Thank 5 you: 6 "We could log into the Horizon servers using our own 7 login details and then use the Riposte system to access 8 the counters -- any changes we made to the counter 9 database would then have our login details attached ..." 10 So in that way, you were using your own log-in 11 details, you were going through the Riposte system to 12 get into the counters and, therefore, any changes would 13 have your log-in details against them; is that right? 14 A. Yes, in the database, from what I recall, if the 15 postmaster was doing transactions, he would be logging, 16 for example, as CTR001. So every line of code in the 17 database would start with CTR001 to identify that 18 postmaster. If we logged in through Riposte, through 19 this way, in my example it may be my code was RWR001, so 20 any transactions or changes I made would start in the 21 database with RWR001. So anybody coming along later 22 would see straight away that it wasn't CTR, it was RWR 23 who had made the changes and put the data in. 24 Q. So there would be a record, an audit trail, as it were, 25 of your actions and what you had done? 70 1 A. Yes. 2 Q. So, to that extent, it's visible and would be apparent 3 to somebody looking, after the event, over who made 4 a relevant entry? 5 A. Yes. The problem with that way of doing things was 6 that, the way the system worked, it would -- if it was 7 then harvesting transactions, it would be looking 8 through and seeing everyone with CTR001. As soon as it 9 came to one that said RWR001 it wouldn't recognise it 10 and there would be errors or it may not process it. It 11 might be that it just skipped them and carried on with 12 the rest of it and didn't flag an error. So then there 13 could be -- if we tried to correct an accounting error 14 or something with the system, it might be that the error 15 wasn't corrected at all and it just skipped it. 16 Q. So, although you might be able to log in and use this 17 route into the system, you might be able to make 18 a correction. By doing -- making the correction, the 19 fix, you were creating one that was either ineffective 20 or could cause other problems? 21 A. Yes. 22 Q. Did you use that method much, then? 23 A. At times it was -- at times that's -- certainly when we 24 knew it wouldn't cause a problem, we would use it. More 25 for when we were doing things, I think, on the actual -- 71 1 either on the routers or the servers themselves and not 2 the counters. 3 Q. Why on the routers and servers rather than counters? 4 A. If we were needed -- I can't remember exactly but 5 sometimes you could change the data as it came into the 6 system or while it was in -- as it came into the servers 7 or while it was already on the servers, in that way you 8 didn't need to go into the counters at the Post Office 9 to change it. 10 Q. Thank you. Can we turn to the second way that you 11 describe, in your (b): 12 "We could log in through Riposte another way, 13 I can't remember the details, in which case it would be 14 difficult to see who had made changes ..." 15 A. Yeah, there was a way of logging in and it wouldn't have 16 a user ID. This is my recollection. It's not 17 necessarily 100 per cent accurate but, from what 18 I remember, then instead of having CTR001 or RWR001, 19 that area would be blank. Again, that would then 20 probably cause processing issues at some point later on, 21 or it may not, depending on which bit of data we were -- 22 was being changed. 23 Q. Why might you use this way? 24 A. I can't remember. I know that it was possible to do it, 25 but I can't remember why it would be done. Maybe it was 72 1 to change actual parameters and not actual data, 2 reference data parameters or something. I can't 3 remember. 4 Q. Again, can we just look back at when you were asked 5 questions on what might be the same topic. I just want 6 to check that they are in your answers given in the 7 Group Litigation Order trial. 8 So can we have up again POL00004074, and go to 9 page 30, please. Go to the bottom right-hand quadrant 10 of the page, which should be internal pagination 116. 11 Can we pick it up, please, at line 22. 12 This is again the Post Office's barrister 13 cross-examining you. He says: 14 "And the second sentence ..." 15 Just so you've got some context here, he's putting 16 part of Mr Godeseth's witness statement to you, okay? 17 He's reading it to you, Mr Godeseth's witness statement, 18 and he says: 19 "And the second sentence: 20 "'The Riposte product managed the message store and 21 it did not allow any message to be updated or deleted, 22 although it did allow for data to be archived once it 23 had reached a sufficient age ...'" 24 You say: "Yes." 25 He asks: "It is correct, isn't it, that Riposte 73 1 didn't allow any transaction line in the message store 2 to be individually deleted or changed or edited in any 3 way?" 4 You replied: "You couldn't do it through Riposte, 5 no. You had to hack the system to do it." 6 Just stopping there, what did you mean by "You had 7 to hack the system to do it"? 8 A. There was another way of running Riposte from -- I can't 9 remember whether it was our counters or from the server, 10 where you could create a session in Riposte, I think it 11 was. 12 Then you could use Riposte to insert data, but then 13 that restricted very much what you could do. So what we 14 were doing, going through the (a) and (b) I've just 15 described, was effectively hacking the system. What 16 they're talking about here is using Riposte to do the 17 stuff for you directly, actually opening up the Riposte 18 session, as it were. So it's like using Microsoft Word 19 or a text editor, but you can either use Microsoft Word 20 to edit a nice document or you could open it up in 21 a text editor, if you knew what you were doing, and do 22 it, you know, through the backdoor, as it were. We were 23 doing it through the backdoor. 24 Q. Why -- 25 A. I don't know if that's -- 74 1 Q. Why were you describing it as a "hack"? 2 A. Because it wasn't the way things were supposed to be 3 done. I don't think it was, anyway. 4 Q. Why was it being done in a way that wasn't supposed to 5 be done? 6 A. Because that was the only way we could get the system 7 back up and running. It was a workaround. 8 Q. Was it just you doing it or were other people in the SSC 9 doing it? 10 A. Everybody was doing it. 11 Q. Was it -- 12 A. (Unclear) 13 Q. I'm sorry, I missed your answer there? 14 A. Yeah, we had unrestricted access. Basically, we could 15 do whatever we wanted. So everybody did it when we had 16 to. 17 Q. Was this known about by your deputy manager of the SSC 18 and the manager of the SSC? 19 A. Oh, definitely, yes. 20 Q. How would they know that everyone in the SSC was doing 21 it? 22 A. Well, they -- it was the other members of the SSC who 23 taught me how to do it. That was the accepted way of 24 doing it, in some instances. 25 Q. Was it reduced to writing, this hack? 75 1 A. Sorry? 2 Q. Was it reduced to writing? Was it written down anywhere 3 that this is the way you do it? 4 A. I don't know. I know that, from things I've read, that 5 there were problems later when the auditors came in and 6 found out we were doing it. So quite possibly. I mean, 7 to start with, I don't think anything was written down. 8 It was all very much flying by the seat of your pants, 9 as it were. 10 Things got written down internally as we went along 11 and then gradually the documentation built up from that. 12 That was one of the problems with the system to start 13 with: that there was no documentation. It was all 14 a scratch -- you know, it was all scratched together, 15 sort of thing. It was a mess. 16 Q. Can we leave this transcript for the moment -- I'm 17 afraid we're going to come back to it in a second and 18 pick up the rest of what you said -- and go to what 19 I think you might be referring to when you said that it 20 was picked up. Can we turn up FUJ00088036. Can you see 21 that this is a document dated August 2002, so it's about 22 halfway or so through your time in the SSC. 23 A. Yeah. 24 Q. It's described as "Secure Support System Outline 25 Design". You're not listed as a contributor or 76 1 a reviewer, nor a person to whom the document would, in 2 due course, be distributed but I just want to ask you 3 a question about a passage in it to see whether it 4 reflects your experience in the SSC. Can we turn to 5 page 15, please. It's under paragraph 4.3.2. It's the 6 "Third line and operational support". It reads: 7 "All support access to the Horizon systems is from 8 physically secure sites. Individuals involved in the 9 support process undergo more frequent security vetting 10 checks. Other than the above controls are vested in 11 manual procedures, requiring managerial sign off 12 controlling access to post office counters where update 13 of data is required. Otherwise third line support has 14 ... 15 The first bullet point: 16 "Unrestricted and unaudited privileged access 17 (system admin) to all systems including post office 18 counter PCs ..." 19 Is that what you were referring to? 20 A. Yes. 21 Q. Is it true that the third line support had unrestricted 22 and unaudited privileged access to all systems, 23 including subpostmasters' counter PCs? 24 A. Yes. 25 Q. Was that widely known within the SSC? 77 1 A. Within the SSC, yes. 2 Q. Was it known, to your knowledge, outside the SSC? 3 A. No. 4 Q. Plainly, by the time of this document it was. 5 A. Yeah, by this time it must have been, but I wouldn't 6 think widely known. I wouldn't think Post Office would 7 have been probably aware of it. 8 Q. Why wouldn't you think Post Office would be aware of it? 9 A. Well, as the customer, I think they would be -- or they 10 should have been -- very concerned, if they were aware 11 that we had that sort of access. At the time I was 12 working there, I just accepted that this was, you know, 13 the practice. It's only since then that I've come to 14 realise that, actually, it's pretty shocking the amount 15 of access we did have. 16 Q. Can we go back, then, to the transcript. POL00004074, 17 page 30. In fact, we'd gone on to page 31. Breaking 18 off, as we had, just after your answer about the hack, 19 you said: 20 "You couldn't do it through Riposte, no. You had to 21 hack the system to do it." 22 Then the Post Office's barrister asks you: 23 "So would this be right, then, that it wouldn't be 24 possible to remotely access a counter and change the 25 data on the message store of that counter remotely?" 78 1 You said: "I believe theoretically, it would." 2 He asked: "How would that be possible? Riposte 3 wouldn't allow you to do it, would it?" 4 You say: "By doing the system I have just said. If 5 you could -- without the message store replicating, so 6 there's no other copies of it, if you could get that 7 message store off, alter the data in some of the lines 8 of code, to do that you would need to strip out all of 9 the preamble and the post-amble, so you're then just 10 left with the basic data as if it had been on the stack 11 or whatever -- forgive me, I'm very rusty on this -- but 12 then by -- I think it was the Riposte import but it 13 might have been something else, you could then re-inject 14 that data which is the process we would have used to 15 rebuild the counter. But if you had changed some of 16 that data, I think it would have rewritten the CRC when 17 it imported it so that when it replicated, the data 18 could theoretically have been changed." 19 Counsel says: "I'm finding it difficult to follow 20 you, and it may be my fault." 21 The judge says: "I follow what the witness is saying 22 but keep exploring it." 23 The Post Office barrister said: "I would like to 24 distinguish though between transactions insertions -- 25 the process of injecting particular transactions into 79 1 the message store, which could be done, with the process 2 of actually manually changing a transaction line that is 3 in the message store and you could insert new 4 transactions, couldn't you, but what you couldn't do is 5 you couldn't edit or indeed individually delete lines 6 that were in the message store itself." 7 You answered: "You'd have to delete all of the 8 message from what I remember. Delete all of the 9 messages down to a certain point to the one you wanted 10 to amend and then inject a load more text or insert more 11 transactions in to make the message store and Riposte 12 think it had been put in by Riposte and by the 13 postmaster." 14 That's where your answer ended. Is what you are 15 describing in that big answer on the page above, between 16 lines 14 on the first page down to line 3 on the second 17 page and then, scrolling down, lines 17 to 22 on the 18 second page -- is that what you were describing in your 19 paragraph (b) in your witness statement, that you could 20 go in another way, in which case it would be difficult 21 to see who had made changes and that this was the hack? 22 A. No, it's what I was describing in paragraph (c). 23 Q. I see. So the paragraph (b) of your witness statement, 24 "We could log in through Riposte another way, I can't 25 remember the details, in which case it would be 80 1 difficult to see who had made the changes"; can you 2 explain to us how that was done, then? 3 A. I can't remember how it was done. I just know that you 4 could do it. The -- you could then have fiddled with 5 it -- for want of a better word -- with the message 6 store but, without the correct user ID at the start of 7 every message, then there would have been errors, things 8 wouldn't have been processed properly, from what 9 I remember. So you wouldn't have gone in that way to 10 make changes to the message store. 11 Q. Okay can we go back to your witness statement, then, to 12 page 6 of the witness statement, and look at (c), the 13 third way. 14 "We could go directly through the communication 15 servers to the [Post Office] gateway and then the 16 counter -- if the [postmaster] wasn't logged in then 17 there would be no ID attached to the database entries, 18 which sometimes caused the batch processing to fail 19 overnight; if the [postmaster] was logged on then any 20 changes we made would have their ID attached -- so as 21 far as the system (and any auditing) was concerned the 22 [subpostmaster] would have been responsible for the 23 transactions." 24 A. That's what I was trying to say. I think that's what 25 I was trying to say in the Post Office transcript we 81 1 just looked at. 2 Q. Thank you. Was this a method that you used frequently, 3 as described in subparagraph (c)? 4 A. We were all pretty adept at it, yeah. 5 Q. Whether you were adept at it -- 6 A. Fairly frequently, yes. 7 Q. Fairly frequently? 8 A. Yes. 9 Q. Okay. Why did you use that method? 10 A. It was the only way to rebuild the counters to get the 11 data off the counters. 12 Q. The footprint that was left would have been the 13 subpostmaster's footprint and not yours? 14 A. Yes. 15 Q. Was there any visibility that you or somebody else in 16 the SSC had done this as opposed to the subpostmaster 17 themselves having done it? 18 A. Sometimes yes, sometimes no. 19 Q. What would distinguish? 20 A. We would -- sometimes it would be recorded. I'm a bit 21 rusty on this now, I'm afraid, but sometimes we told the 22 postmaster we were going to do it. While we were doing 23 this, the postmaster couldn't use the counter. It was 24 very important that nobody used it. At other times, 25 especially if maybe the postmaster -- I'm just thinking. 82 1 I'm just trying to remember something else. I was going 2 to say if the postmaster had gone to lunch, for 3 instance, we could have gone in and done things without 4 him knowing. There may have been a way to put the data 5 in at the counter while the postmaster was actually 6 logged off. There may have -- I can't remember exactly 7 but there may have been a way to fool the counter into 8 thinking that the postmaster that logged on to do it. 9 I can't fully remember that. 10 Certainly, we were on occasion asked -- I can't 11 remember the details. I know that since the court case, 12 it may be during the court case, I saw documentation to 13 the effect that we had at times gone into the counter 14 without the postmaster or even POL knowing to make 15 changes to the data and, in the way that I'm talking in 16 item (c) here. So the postmaster may have logged on and 17 gone to lunch and left the computer logged on, so then 18 we went in, made the changes we needed to fix the 19 problem, and then logged out again, leaving the 20 postmaster completely unaware that we'd done it. 21 Q. Can we go over the page on your witness statement, 22 please, to look at the security protocols about 23 accessing subpostmasters' systems, and look at 24 paragraph 15. You say: 25 "The Inquiry has asked about security protocols 83 1 regarding access to [subpostmasters'] systems. I don't 2 remember any security protocols; we sometimes connected 3 to [post office] counters without the postmaster being 4 aware that we were 'looking over their shoulder'. In 5 the early days, I frequently logged on to counters to 6 see what was happening; there was no record of my doing 7 so but I think this changed after I had left." 8 A. Yeah. 9 Q. Can we look, please, at the transcript again of the High 10 Court trial. That's POL00004074, page 33, please. It's 11 the bottom left hand quadrant and it's line 22, the foot 12 of the page. This is partway through your answer. You 13 say: 14 "In circumstances we could do that. In other 15 instances, the way I remember it is that for the system 16 to operate correctly for the accounting, it had to be 17 the same user ID logged on, so that the postmaster or 18 that clerk or whatever would have to be logged on with 19 their ID and password so that any data we changed or put 20 back on would then go in with their ID, which is why 21 they couldn't use it. Then that data would be picked up 22 correctly by Riposte. Riposte would assume that the 23 postmaster had been operating as normal and would accept 24 the data into the message store and process it 25 correctly." 84 1 Question: "Could you tell me what were the 2 circumstances in which you had to use the same ID as the 3 original user?" 4 Answer: "I can't remember what the differences were 5 for the different errors but it depended on what error 6 was coming up and what bit of data was corrupt, where 7 the corruption lay in the message store." 8 Question: "So you can't think of a specific reason 9 why it would have to be the same person but you're 10 saying that it did sometimes?" 11 Answer: "Yes, it -- sorry. I didn't let you finish. 12 I've lost my train of thought now, sorry. It often made 13 it much cleaner for accounting reasons. From what 14 I remember, if it was the same user ID, all of this, all 15 of these actions would be detailed in the PinICL and if, 16 from what I remember, if you were accessing the counter 17 in this way, two people had to be there, one was 18 an independent witness, to make sure that everything was 19 going correctly." 20 Just stopping there, are you describing here that 21 there needed to be two people, one of whom was 22 an independent witness, to witness the hack that you've 23 described? 24 A. If you were making changes to the database and putting 25 data in, then yes. They would watch as you went through 85 1 all the steps to clean the data up, just to -- to double 2 check to make sure you hadn't made a mistake and deleted 3 something in error. I think that stemmed from an issue 4 when at some point somebody did make an error and it 5 really messed up the processing later. So that was 6 a lesson learned as we were going along. Again, my 7 memory is very hazy on this but I think that's why it 8 was that we then employed two people to make sure that 9 there weren't mistakes. 10 Q. Then scrolling down, counsel asked you: 11 "So there would have to be what we now call PEAKs 12 and there would have to be two pairs of eyes?" 13 You say: "That was what --" 14 Then he carried on: "It would never be left to one 15 particular member of the SSC to do it on his own?" 16 Answer: "It was never supposed to be, and I don't 17 think it ever was, but I'm not sure." 18 Question: "So this a formal process then, is it?" 19 You answered "Yes." 20 "Which the SSC took very seriously?" 21 Answer: "It was developed and taken very seriously, 22 yes." 23 So did the position change over your time within the 24 SSC? 25 A. I think so, I can't really remember. Also, reading the 86 1 next line from 16 to 19, that's where, at that point, 2 reading this, I was -- I believed that formal consent 3 from the Post Office was required. It was after this 4 that I saw the documentation that contradicts this and 5 there's actually times when the Post Office weren't 6 informed. 7 Q. So carrying on reading then from 14, just to get the 8 question, he asked you: 9 "Is it also the case that the Post Office consent 10 was always needed for this kind of process?" 11 You said: "I was there we were supposed to speak to 12 the postmaster to get his consent, so from Post Office 13 consent, that's what I believe you mean by that. Formal 14 consent from the Post Office itself, maybe not." 15 Just stopping there, did you always speak to the 16 postmaster to get consent? 17 A. From what I remember now, no. But memory is a funny 18 thing and sometimes, after this length of time, you 19 remember things that didn't actually happen. So I can't 20 completely, hand on heart, say that that's true or not. 21 Q. You mentioned that you had seen something after giving 22 evidence here that had maybe changed your view. What 23 was that? 24 A. There was some documentation that came up right at the 25 end of my interrogation. It may have been right at the 87 1 end, it may have been right after I left, I saw it and 2 it was a statement, I think it was from Mik Peach, 3 saying that we weren't to inform the Post Office of 4 this, this particular item. I can't remember exactly 5 what it was. 6 Q. Other than the postmaster themselves, do you remember 7 any communication, written or verbal, between you and 8 other members of the SSC team and Post Office managers 9 or somebody within security within the Post Office, or 10 something like that, before you undertook this exercise, 11 this hack? 12 A. No. The only person we would have spoken to for any 13 authorisation would have been Mik, Mik Peach, the 14 manager. Everything came through him, really. 15 MR BEER: Thank you. We can take that down now. 16 Sir, I wonder whether we might unusually ask for 17 an earlier lunch today. There's been some quite heavy 18 transcript that I've gone through with Mr Roll and 19 I suspect that he, but also me, would appreciate a break 20 so can we break for lunch? 21 SIR WYN WILLIAMS: Of course. 1.45? 22 MR BEER: Yes, thank you very much, sir. 23 SIR WYN WILLIAMS: Fine. See you then. 24 MR BEER: Thank you. 25 (12.45 pm) 88 1 (The Short Adjournment) 2 (1.45 pm) 3 MR BEER: Good afternoon, sir. Can you see and hear me? 4 SIR WYN WILLIAMS: Yes, I can. 5 MR BEER: Mr Roll, can you see and hear me? 6 A. Yes, I can. 7 Q. Thank you. Can we have up on the screen, please, 8 POL00000900. Thank you very much. Can you see the 9 title of this document, it's at the top of the page, 10 Mr Roll, and under "Document title", "CS Support 11 Services Operations Manual"? 12 A. Yeah. 13 Q. It's dated in this version, version 2, 29 January 2001. 14 A. Yes. 15 Q. That would have predated your arrival in the SSC, 16 wouldn't it? 17 A. Yes. 18 Q. We've got earlier versions of this in 1999 and 2000 but 19 I'm going to show you this one because it's most 20 proximate to your arrival in the SSC. Can you see in 21 the middle of the page or the bottom of the page there 22 it says, "Owner: Peter Burden"; do you remember who he 23 was? 24 A. No. 25 Q. At the foot of the page under "Distribution", the 89 1 distribution includes, as the third person there, the CS 2 Support Services Manager. Would that be Mr Peach? 3 A. Yes. I think. 4 Q. Which one of those would be Mr Peach? Would it be the 5 second one from the bottom, SSC Manager? 6 A. Yes, I believe so. 7 Q. You'll see that, amongst the distribution lists, is 8 "Pathway document library". Do you remember what that 9 was? 10 A. No. 11 Q. How were documents like this, this is an operations 12 manual that's about the SSC, promulgated or distributed 13 to people like you that were working essentially on the 14 floor of the SSC? 15 A. I don't know. I can't remember ever seeing this. 16 Q. Was there an intranet, part of which had a repository 17 for policy and other documents? 18 A. I don't think so. I can't remember. 19 Q. Can we just look at this, please, because it describes 20 a process that may be relevant to the issue we were 21 discussing before lunch. Can we go, please, to 22 page 39 -- sorry, not page 39 -- page 15, and scroll 23 down to paragraph 4.3. This document says, I'll read it 24 through slowly: 25 "The SSC has access to the live system which can be 90 1 used to correct data on the system when this has been 2 corrupted in some way." 3 That accords with the evidence you have given so 4 far, doesn't it? 5 A. Yes. 6 Q. "The procedure for doing this is as follows: 7 "The originator of the change: 8 "1. Completes an Operational Correction Request 9 (OCR) form for every change to data on the live system. 10 "The originator may be anyone within ICL Pathway but 11 it is normally the Duty Manager, or a Problem Manager or 12 Business Support Manager when an incident or problem has 13 been caused by an error in the data. It can be 14 completed by an SSC staff member who detects that the 15 data in the system has become corrupted in the course of 16 diagnosing a fault." 17 Then "The originator of the change": 18 "2. Emails the OCR form to an authoriser, 19 electronically signing it where possible, and where this 20 is not possible, telephoning the authoriser to confirm 21 that they are sending an OCR [an Operational Correction 22 Request]." 23 Can we turn to page 39 of the document, please, 24 where we see an OCR form or a template OCR form, 25 an operational correction request. You'll see there's 91 1 a title, who the OCR was raised by, their location, when 2 they raised it, the type of change requested, the system 3 to be changed, a due date, and an associated PinICL 4 number, an authorisation signature, date and position 5 and then, scrolling down, at the foot of the page, 6 "Purpose and details of the change". 7 Then over the page, please, "Regression path", and 8 then "Signature of the SSC", person who did the work, 9 their printed name, the witness, either somebody in the 10 SSC or the fourth line signature, and then name, 11 completion date, "Was change tested on reference rigs 12 prior to application, yes or no", system state before 13 change, system state after change and then, scrolling 14 down, "Comments". 15 Is that a document with which you were familiar? 16 A. I don't remember seeing it. I can't remember it. 17 Q. Was this something that you, to your memory, had to fill 18 out or somebody had to fill out before they requested 19 you to make a change, it had to be countersigned. You, 20 when making the change, had to sign it and it had to be 21 countersigned by the witness. Was that habitually done? 22 A. I can't remember at all seeing one of these documents 23 before. 24 Q. If we go back to page 15 of the document, please, at the 25 foot of the page. We had got to paragraph 2 and then it 92 1 continues: 2 "The authoriser must be one of the following: 3 "Duty Manager 4 "Business Support Manager 5 "CS Operations Manager 6 "SSC manager 7 "Release manager." 8 Then: 9 "The authoriser: 10 "1. Authorises the change, or reports back to the 11 originator why they are not authorising the change. 12 "2. Forwards the OCR form to the SSC electronically 13 with an encrypted electronic signature file. 14 "The SSC staff member who is to perform the change 15 [I think that would be you]: 16 "3. Checks the electronic signature of the 17 authoriser. 18 "4. Stores the OCR form and the signature file in 19 the received OCRs folder on the SSC server. 20 "5. Wherever possible, produces a script to make 21 the data change and test the script on the SSC reference 22 rig prior to running it on the live system. 23 "6. Completes the relevant sections on the OCR form 24 to confirm whether they have produced and tested 25 a script or not. 93 1 "7. Prior to making the change ... documents the 2 state of the affect part of the system and completes the 3 regression path details on the OCR form ... 4 "8. Makes the change on the live system. 5 "At least two people must be present when making 6 changes to the live system. Normally these are SSC 7 staff, but can be one SSC staff member and one person 8 from the fourth line support unit responsible for the 9 area in which the data change will take place, or one 10 SSC staff member and one OSD staff member 11 "9. On completing the data change, documents the 12 state of the affected part of the system and mails 13 an electronically signed copy of the OCR form to the 14 second person who was present whilst making the change. 15 "10. The second person also electronically signs 16 the form and emails it to either the SSC manager or the 17 SSC website controller. 18 "11. Updates the PinICL and reports back to the 19 originator to confirm that the change has been 20 completed." 21 Then: 22 "The SSC Manager or SSC website controller: 23 "12. Checks the electronic signatures. 24 "13. Files the OCR in the complete OCR folder on 25 the SSC server." 94 1 That very involved and complicated process, was that 2 something that was habitually done when you were 3 undertaking the category C hacks that you described 4 earlier? 5 A. I don't remember doing that, no. We might have done. 6 I just can't remember. I do think that, on occasion, 7 we'd made changes without a PinICL being raised. 8 Q. In your answer there, you said that you don't remember 9 whether it was done or not, and it might have been. Is 10 that the category of memory that we're dealing with 11 here, you're saying this could perfectly well have been 12 undertaken on each occasion, you simply now don't 13 remember it? 14 A. That's right, yeah, I don't think it was undertaken. 15 I don't remember it being undertaken at all but I can't 16 remember. 17 Q. Thank you very much, that can be taken down from the 18 screen now. 19 Can we look, please, at POL00023432. This is 20 an email exchange in 2008, so four years after you left, 21 in which you were not involved. If we just look at the 22 second page, please. The Inquiry is familiar with this 23 but I just ought to give you some context first. It 24 concerns a subpostmaster saying to somebody on his area 25 that, on a number of occasions, figures have appeared in 95 1 the cheque line of his account. 2 Do you see, in numbered paragraph 1, he, the 3 subpostmaster, that's Mr Graham Ward: 4 "... claims that on a number of occasions figures 5 have appeared in the cheques line of his account. He 6 suspects that these have been input into his account 7 electronically without his knowledge and consent. He is 8 certain that he has cleared and remmed out cheques in 9 the right way and tells me that cheques must be properly 10 cleared on the system to progress to a new account. 11 Then paragraph 2: 12 "He has made good about £10,000 and not made good 13 about £11,000 of the shortages which arise from these 14 figures. He claims that because of the abnormal nature 15 of these entries, the shortages have not just rolled 16 over from one branch trading statement to the next, but 17 have accumulated -- each being added to the last (eg if 18 the account in period one showed a shortage of £100 19 which was not made good, then the shortage shown in 20 period 2 would be £200)." 21 So it was doubling up, week on week. 22 Had you heard of any similar issue when you were 23 working in the SSC? 24 A. I have heard of similar issues but I can't remember if 25 I remembered them from when I was there or whether it's 96 1 things I have read since in the reports about the court 2 case, and so on and so forth. 3 Q. So, very fairly, you're distinguishing between your real 4 memory of events in which you were involved, and things 5 which you read and heard afterwards? 6 A. Yeah, I can't remember whether I was aware of them at 7 the time. 8 Q. Can you help us: what would be the investigation that 9 would be required within the SSC, if this kind of report 10 had been made to you? 11 A. Other than going through the transactions and the number 12 of cheques -- to look at the number of cheques coming in 13 and out and trying to work out where the figures were 14 being generated, using extracts from the database 15 I can't really remember much more about that, apart from 16 that. 17 Q. Would you examine the code that has undertaken the task 18 of putting in cheques in and out? 19 A. I think we would only have gone to examine the code if 20 we could find exactly where in the message store the 21 problem was occurring. The problem is that there is so 22 much code that, if you don't go in without -- if you go 23 in without any sort of reference point, it would be like 24 trying to find a paragraph or an apostrophe in War and 25 Peace. You wouldn't know where to start. You'd need to 97 1 know where in the data store, what was going on from the 2 data store, from the database, to give you some idea as 3 to where to start looking. That's what I remember. 4 Q. So how, practically, would -- if this is all you've got 5 to go on, how practically would you go about it? 6 A. First step would be to download messages from the 7 message store and start trying to follow through what 8 the figures were and try to work out what was happening 9 with the figures. 10 Q. Would you be able to see from the message store this 11 doubling up that the subpostmaster is referring to? 12 A. You would probably see that the figures had doubled. 13 Whether you would be able to work out exactly why, I'm 14 not sure. My memory -- it's a long time ago and I just 15 can't remember, I'm afraid. 16 MR BEER: Mr Roll, thank you very much. They're the only 17 questions that I ask you. There may be some other 18 questions from other Core Participants. Can I just 19 check? 20 MR STEIN: Sir, I may like to ask a question of Mr Roll. 21 I just need to take instructions. May we have 22 a five-minute break now before I do so? 23 SIR WYN WILLIAMS: Of course. 24 MR STEIN: Or leave it to my learned friend's who may want 25 to ask questions on behalf of their own clients to carry 98 1 on and deal with that separately. 2 SIR WYN WILLIAMS: Let's just check, are there other people 3 who want to ask questions? 4 MS PAGE: I do want to ask some short questions. I don't 5 imagine they'll take very long. 6 SIR WYN WILLIAMS: Let's just have a five-minute break, then 7 and then we'll have all the questions sequentially. 8 MR BEER: Thank you, sir. 9 (2.03 pm) 10 (A short break) 11 (2.12 pm) 12 MR BEER: Can you see and hear us again? 13 SIR WYN WILLIAMS: I can. 14 MR BEER: I think we're now ready, if Mr Roll is ready for 15 questions from Mr Stein. 16 SIR WYN WILLIAMS: Yes. 17 Questioned by MR STEIN 18 MR STEIN: Sir, first of all thank you for the time. It has 19 been of assistance in narrowing down the focus of any 20 questions. 21 Mr Roll, I appear on behalf of a large number of 22 subpostmasters and mistresses. I've got one question to 23 ask you arising out of the questions asked by Mr Beer 24 this morning and this afternoon. You've discussed with 25 Mr Beer what can happen when a computer system being 99 1 used by a subpostmaster had a dodgy on/off button; do 2 you remember that? 3 A. Yes. 4 Q. Now, you explained in your evidence, I believe this 5 morning, that that could lead and did lead to a loss of 6 data integrity within that system being used by that 7 branch; is that correct? 8 A. Yes. 9 Q. Can we therefore assume that losses of power would also 10 have the same effect, in other words losing data 11 integrity within that particular branch system? 12 A. I suppose there is the potential, the big loss of 13 integrity with this particular instance was that, from 14 my recollection, although the data was on the computer 15 it wasn't getting through to the systems or not getting 16 through correctly, so then it wasn't being processed 17 correctly. 18 Q. Right, so there's a potential for losing data when the 19 power goes but there's also, in your recollection, 20 difficulties with data integrity when there's 21 a connectivity issue; is that correct? 22 A. Yes. 23 Q. Okay. Now, if the branch loses connectivity with the 24 rest of the Horizon System for, I don't know, cable 25 reasons or some other hardware reason, can that lead to 100 1 the same problem in isolating that branch from the rest 2 of the system and losing data? 3 A. The data would hopefully still be on the computers in 4 the branch but, certainly from an estate or from the 5 server perspective, it would be as if that Post Office 6 had closed and was not operating. So the data wouldn't 7 be visible from the servers until it was switched back 8 on and the data had managed to replicate through 9 overnight. 10 Q. Right. Would it always recover? We know that it was 11 meant to, but would it always? 12 A. If there was a power failure or if it had suddenly been 13 cut off then there's always I suppose the potential 14 damage, hardware damage to the disk or some of it -- 15 maybe boards in the computer. So, from that 16 perspective, if the computer was made inoperable then 17 you would lose the data. So those transactions would 18 have been lost. There's the potential, then, that if 19 you can't recover any of the data from the disk, then 20 without a paper audit trail, you wouldn't know what had 21 gone on in the counter -- in the post office that day. 22 Q. Okay, my last question on this: were these problems, to 23 your recollection, explained to subpostmasters and 24 mistresses as being a potential difficulty that could 25 lose data? 101 1 A. I don't know, I don't remember, well I don't remember 2 ever explaining that, and I'm pretty sure I didn't. 3 Whether anybody else on the installation teams had 4 explained this to postmasters, I don't know. 5 MR STEIN: Thank you, Mr Roll. 6 Questioned by MS PAGE 7 MS PAGE: Mr Roll, it's Flora Page here, also representing 8 a number of the subpostmasters. You mentioned earlier 9 the system or tool known as Tivoli, which dealt with the 10 automated system driven alerts; is that right? 11 A. Yeah, but I can't remember much about that at all, I'm 12 afraid. 13 Q. No, okay. Just one question, then: do you remember 14 there being any sort of routine or process around it or 15 were those alerts just dealt with in exactly the same 16 way as a call coming in from an SPM? 17 A. I can't remember, I'm afraid. 18 MS PAGE: All right. Thank you very much. 19 Further questioned by MR BEER 20 MR BEER: Sir, I think those are all of the questions from 21 everyone. There's just one thing that I'd like to do 22 before Mr Roll finishes giving his evidence, and ask for 23 POL00000678 to be displayed. You mentioned, Mr Roll, 24 earlier, that you'd made three witness statements not 25 two, when I -- 102 1 A. Yes. 2 Q. You finally corrected me. I just wanted to show you the 3 third one. It was, in fact, an amended version of the 4 second one; is that right? 5 A. I can't remember. It may well have been, yes. 6 Q. We've got this on the screen now and we can see that the 7 amendments to it are in red. You can see that in the 8 tramlines, and we can see a further date of it, if we go 9 to the last page, page 8., and scroll down. We can see 10 where you re-signed it. 11 A. Yes. 12 Q. It'll have "GRO" on it. That's covering up your 13 signature so people can't use it. 14 A. Yes. 15 Q. There was only one passage in this witness statement 16 that I took you to that's the subject of amendment, and 17 so I should take you back to it and ask you about the 18 amendments. So can we look, please, at the bottom of 19 page 2 and scroll down, please, to paragraph 8. Can you 20 see paragraph 8 with the amendments in red? You're here 21 also dealing with the laptop standby/switch-off power 22 issue and what happened when you raised it with your 23 manager. At the foot of the page, in your amended 24 statement, you say: 25 "When I raised this with my manager Mik Peach, he 103 1 initially told me not to do anything until he had spoken 2 to someone about this. Mik did subsequently talk to the 3 hardware team, at which point I found out this was 4 a known problem ..." 5 Then at the end, you added the sentence: 6 "I was told by Mik Peach not to include any details 7 of this when I closed the PinICL." 8 Were the amendments that you made to your witness 9 statement, in one case amending an order of events, and 10 in the second by adding a sentence at the end, correct? 11 A. They were correct, yes. I think I was trying to clarify 12 it. 13 MR BEER: Yes. Thank you very much. They're the only 14 questions that I ask you, Mr Roll. Thank you. 15 SIR WYN WILLIAMS: Mr Roll, I'm very grateful to you for 16 giving your evidence to the Inquiry. As is obvious to 17 everyone, you have been involved in providing assistance 18 to those who are seeking to unravel what has occurred 19 over very many years, and I don't think I can do other 20 than thank you for that, as well. So on many fronts, 21 thank you very much, Mr Roll. 22 THE WITNESS: Thank you. 23 MR BEER: Sir, that's it for our witnesses this week and 24 indeed this month. Our next hearing I think is 25 scheduled to occur on 27 April in relation to 104 1 compensation issues, perhaps including bankruptcy and, 2 depending on the position as it's developed by then, tax 3 issues. So that is our next hearing date. 4 SIR WYN WILLIAMS: I know that you're right, Mr Beer. Can 5 I say that, whereas in the previous hearings I have in 6 effect invited the parties who wish to make written 7 representations and speak to say whatever it is that 8 they want to say, this hearing may be a little more 9 focused in that I might well take the lead in 10 determining what should be the subject of either written 11 or oral submissions. In any event, over the course of 12 the coming couple of weeks, I hope, I will issue formal 13 directions as to how we intend to proceed. 14 MR BEER: Thank you very much, sir. 15 SIR WYN WILLIAMS: All right, then. If not before, 16 27 April. 17 MR BEER: Thank you, sir. 18 (2.22 pm) 19 (The hearing adjourned until 27 April 2023) 20 21 22 23 24 25 105 I N D E X RICHARD WILLIAM ROLL (affirmed) ......................1 Questioned by MR BEER .........................1 Questioned by MR STEIN .......................99 Questioned by MS PAGE .......................102 Further questioned by MR BEER ...............102 106