1 Tuesday, 18 October 2022 2 (10.00 am) 3 MR BEER: Good morning, sir. Can you see and hear me? 4 SIR WYN WILLIAMS: Yes, I can. Can you hear me? 5 MR BEER: Yes. Thank you very much. Can I call Charles 6 Cipione, please. 7 SIR WYN WILLIAMS: Yes. 8 CHARLES CIPIONE (sworn) 9 Examined by MR BEER 10 MR BEER: Thank you very much, Mr Cipione. Please take 11 a seat. Can you give us your full name, please? 12 A. Charles Anthony Cipione. 13 Q. You've been instructed as an expert witness by this 14 Inquiry and you've kindly prepared a written report as 15 a result of those instructions; is that right? 16 A. Yes. 17 Q. Please can we see your report of 14 September 2022. 18 Thank you very much. For the record, it's EXPG0000001. 19 You've got a hard copy in front of you, in case you need 20 it, but it should be displayed on the screen as well. 21 Is that the first page of your report? 22 A. Yes. 23 Q. Including appendices, is it 174 pages in length? 24 A. Yes. 25 Q. I think it's divided into two parts, Parts 1 and 2, and 1 1 Part 1 is between pages 12 and 64, if you could just 2 turn that up and confirm, please? 3 A. Yes. 4 Q. So that's Part 1 beginning on page 12 and then, if we go 5 forward to 64, we can see the end of Part 1, and then 6 Part 2 is between pages 65 and 160 -- 7 A. Yes. 8 Q. -- when the appendices begin on page 161 and I think 9 there are five appendices, A, B, C1, C2, and C3? 10 A. Yes. 11 Q. As you know, you're kindly giving evidence to us, 12 Mr Cipione, in two stages, today and possibly tomorrow, 13 about the matters addressed in Part 1 of your report, 14 and then on 17 and 18 November this year you will return 15 to give evidence about Part 2 of your report; do you 16 understand that? 17 A. Yes. 18 Q. Can you confirm the following, please: firstly, you've 19 made clear in Part A (sic) of your report, which facts 20 and matters referred to are within your own knowledge 21 and which are not. 22 A. Yes. 23 Q. Secondly, that those facts in Part 1 of your report that 24 are within your own knowledge, you confirm to be true? 25 A. Yes. 2 1 Q. Thirdly, that the opinions you've expressed in Part 1 of 2 your report represent your true and complete 3 professional opinions on the matters to which they 4 refer? 5 A. Yes. 6 Q. Thank you. Can I begin by asking you your 7 qualifications and experience as an expert witness. 8 I think you hold a Bachelor of Science degree from 9 Texas A&M; is that right? 10 A. That is correct. 11 Q. The A&M standing, I think, for "agricultural and 12 mechanical"? 13 A. That is correct. 14 Q. You hold a Masters of Business Administration, an MBA, 15 from the same university? 16 A. That is correct. 17 Q. I think your professional career began at Arthur 18 Andersen, a firm headquartered in Chicago and which, at 19 its height, was one of the largest public accounting 20 firms in the 1990s with 85,000 employees operating 21 across the world; is that right? 22 A. Yes. 23 Q. Is it right that you worked in the information systems 24 risk management business within the firm? 25 A. Yes, that is correct. 3 1 Q. What did your work then involve? 2 A. The work that we performed while I was at Arthur 3 Andersen was around systems controls, a lot of general 4 controls reviews which simply means making sure that the 5 proper policies and procedures surrounding the financial 6 systems at firms were in place and allowed for the 7 auditors to rely upon the information that was in those 8 systems. Also performed a number of application 9 controls tests, which was a little bit more in-depth, on 10 individual applications, whether they were efficacious 11 or not, and various -- it really depended on which 12 project I was working on but it was generally to look at 13 the efficacy of systems. 14 Q. You say in your report you developed and implemented 15 database applications and analyses relating to 16 litigation and bankruptcy clients? 17 A. That is correct. In addition to the audit projects that 18 I've just mentioned, I also developed -- also worked 19 a lot with our litigation and our bankruptcy group in 20 developing, maintaining -- developing, deploying and 21 maintaining database applications related to either 22 companies that we had been hired to help through the 23 bankruptcy process in the United States or companies 24 that hired us to perform expert work in litigation -- in 25 the litigation arena. 4 1 Q. I think, in due course, you left Arthur Andersen and set 2 up your own consulting firm, Cipione & Associates? 3 A. That is correct. 4 Q. When was that? 5 A. That would have been around 1994. 6 Q. What did that venture involve? 7 A. That was -- basically I was a software developer. 8 I was -- for various clients, I would design, develop, 9 deploy and maintain software applications. 10 Q. You mentioned in your report, for the notice 2.1.4, that 11 the software you designed, developed and maintained was 12 DOS. What is or was DOS? 13 A. DOS stands for Disk Operating System. It's basically 14 the operating system for PC-based computers that was 15 developed by Microsoft. 16 Q. It runs from a disk drive; is that right? 17 A. Yes, that's correct. 18 Q. This was a Microsoft product. The predecessor to 19 Microsoft Windows; is that right? 20 A. That's correct. 21 Q. I think in 2001, you joined AlixPartners; is that right? 22 A. That's correct. 23 Q. Can you explain, please, who or what AlixPartners is? 24 A. AlixPartners is a global consultancy that performs 25 a variety of services for clients, probably at that 5 1 point in time was best known for their turnaround or 2 restructuring services within the United States. They 3 basically would take over companies that offered 4 Chapter 11 protection within the United States and 5 operate those companies through the bankruptcy process, 6 which often terminated in a plan of confirmation to get 7 those companies out of bankruptcy. 8 Q. Thank you. You say that when you joined you helped 9 establish the claims management service; what did that 10 involve? 11 A. Yes, so a big part of the bankruptcy process in the 12 United States has to do with the reporting of all of the 13 assets and liabilities of the debtor company that's 14 going into bankruptcy, for purposes of allowing the 15 creditors to understand the debtor's position on amounts 16 that they think that they owe the creditors. 17 There are a number of -- there are several reports 18 that are required by the court, AlixPartners did not 19 have a system to do that in the group that I belonged 20 to. As we were acquired by AlixPartners, I took over 21 all of the administrative responsibilities for reporting 22 in the court. 23 Q. Thank you. You say this involved interrogating, 24 collecting and organising vast amounts of disparate, 25 financial and operational data from your client systems; 6 1 is that right? 2 A. That is correct. 3 Q. For what kind of clients was that service established 4 and operated? 5 A. So that's the bankruptcy process that I was just 6 referring to. Some example clients that we worked on 7 were WorldCom, General Motors, Kmart, there are a vast 8 number of them, but all very large, very major 9 bankruptcies from the 2001 to current time frame. 10 Q. Were you the architect of those systems? 11 A. Yes. 12 Q. Are they still in use today? 13 A. They are. 14 Q. I think you're presently a managing director within the 15 risk analytics group at AlixPartners; is that right? 16 A. That's correct. 17 Q. You've held that position for over 15 years? 18 A. Yes. 19 Q. Is it right you've been retained in that position by 20 clients to provide factual and expert evidence in 21 relation to the efficacy of application systems and the 22 management and analysis of datasets relating to 23 litigation and regulatory issues? 24 A. Yes. 25 Q. I think, although it's right that you've plainly given 7 1 expert evidence before, it's fair to say that you are 2 primarily a practitioner rather than somebody who spends 3 most of their time in the courts; is that right? 4 A. That is correct. 5 Q. Overall, therefore, you have some 30 years' experience 6 in information technology; is that right? 7 A. Yes. 8 Q. Can I turn secondly to look at your instructions. 9 You've been given two sets of instructions by the 10 Inquiry legal team. The first of them provided to 11 you -- I'm at paragraph 2.3 -- on 2 June 2022 and then 12 addendum instructions on 27 July 2022; is that right? 13 A. Yes. 14 Q. Are those instructions fairly summarised -- can we 15 display this, please -- paragraph 2.3.3 of your report, 16 which is at page 7? 17 Towards the foot of the page, 2.3.3. Does that, in 18 paragraph 2.3.3, in (a), (b) and (c), fairly summarise 19 your instructions -- 20 A. Yes, it does. 21 Q. -- namely an introduction to the Horizon System and 22 other key terms that will seek to assist this Inquiry in 23 understanding the substance of your report and other 24 submissions that might be made to the Inquiry. You were 25 instructed that the introduction to the Horizon System 8 1 should be tailored so as to be understandable to the 2 Inquiry, the Core Participants to the Inquiry, to 3 members of the public who may not have prior knowledge 4 of the Horizon IT System; is that correct? 5 A. Yes, that is correct. 6 Q. Is that essentially Part 1 of your report? 7 A. That is part 1, yes. 8 Q. You were instructed to analyse and identify -- sorry, 9 and illustrate any themes in the problems that were 10 being experienced by users, in the period up to and 11 including the rollout of the Horizon IT System, 12 including how those problems were resolved or escalated, 13 and the key individuals who were involved in these 14 processes; is that essentially Part 2 of your report -- 15 A. Yes. 16 Q. -- taken together with the third thing: any overall 17 observations or conclusions, that were within your 18 professional expertise, as to the themes that you 19 identified and the potential reasons for them? 20 A. Yes. 21 Q. You say in your report that, although those were your 22 instructions and therefore provided the basis for the 23 determination of the scope of your work, you've 24 nonetheless been responsible as an independent expert 25 for developing your own approach to the questions posed 9 1 by the instructions; is that right? 2 A. Yes, that is correct. 3 Q. Did you undertake your review of the material between 4 June and September this year -- 5 A. Yes. 6 Q. -- assisted by a team from AlixPartners, including 7 colleagues in the United Kingdom? 8 A. Yes. 9 Q. They've assisted you, I think, with anglicising some of 10 the phrases, alternative phrases in the report? 11 A. Indeed. They did. 12 Q. Thank you, so spelling words like "colour" -- 13 A. Yes. 14 Q. -- and "defence" and things like that, presumably? 15 A. Yes. 16 Q. Okay. In terms of the materials relied on, are they 17 listed over pages 161 to 165 of your report? That's 18 appendix H, 161-165. 19 If we could turn those up, please. Thank you. 20 A. Yes. 21 Q. We can see on page 161 a list of PinICLs or PEAKs, and 22 then over the page, please, to 162 we can see the list 23 of PinICLs and PEAKs continued; is that right? 24 A. That is correct. 25 Q. For now, I think, it's sufficient to know that a PinICL 10 1 was a customised incident logging and resolution 2 tracking system initially adopted by Fujitsu between 3 I think '96 and 2003? 4 A. That's correct. 5 Q. We'll come back to these in detail later. Then the PEAK 6 was the customised incident logging system designed to 7 replace PinICL in, I think, 2003? 8 A. Yes. 9 Q. These PinICLs and PEAKs were a selection taken, I think, 10 from some 55,000 such documents that you were provided 11 with? 12 A. Yes. 13 Q. I think, as you tell us in the report, you used computer 14 assisted technology to review that material; is that 15 right? 16 A. That is correct. 17 Q. Because what you describe as a "brute force approach" 18 wasn't possible and was inadvisable with that volume of 19 data. By "brute force", in this context, do you mean 20 reading and analysing every one of 55,000 error logs? 21 A. That is exactly what I mean. 22 Q. Rather than the algorithm Brute Force? 23 A. That's right. 24 Q. Okay, got it. Then on 162 you continue by listing some 25 monthly reports from Pathway and ICL Pathway? 11 1 A. Yes. 2 Q. Then if we go to 163, please. We see those reports 3 continue, and then a list of other background materials 4 that you had regard to. Then over the page to 164 and 5 165, some publicly available materials that you list? 6 A. Yes. 7 Q. Your work and therefore the observations and conclusions 8 in the report, and the evidence you'll give today, are 9 based only on the documentary evidence and data provided 10 to you by the Inquiry, which in turn was provided by 11 some of the Core Participants; is that right? 12 A. Yes. 13 Q. That was primarily Fujitsu; is that right? 14 A. That is correct. 15 Q. Primarily in the period from July '96 to December 2000? 16 A. Yes. 17 Q. Thank you. Can I turn thirdly to the scope of Part 1 of 18 your report and the evidence that's to be given today. 19 Your instructions relate specifically to Phase 2 of the 20 Inquiry, and this is paragraph 2.4.2 of your report, and 21 therefore address the procurement, design, pilot, 22 rollout and modification of and to the system? 23 A. Yes. 24 Q. Like this Inquiry, you've adopted the umbrella term 25 "Horizon System" and "Horizon IT System" that was 12 1 employed by Mr Justice Fraser in his Horizon Issues 2 judgment -- 3 A. Yes. 4 Q. -- which is: 5 "... the Horizon computer system hardware and 6 software, communications equipment in branch and central 7 data centres where records of transactions made in 8 branches were processed." 9 Is that right? 10 A. That is correct. 11 Q. Now, I think Part 1 of your report is itself divided 12 into four parts. In section 3 of your report, or the 13 paragraphs beginning with 3, you address the theory of 14 system design and development? 15 A. That is correct. 16 Q. Just tell us why is that important, the theory of system 17 design and development? 18 A. Understanding, especially for people who are not 19 familiar with the intricacies of system design 20 development, deployment and maintenance, it's important 21 to have just a general overview of what goes into that 22 and what to expect from that process. So I felt it was 23 important to just spend a little bit of time in my 24 report to explain some concepts that I feel will be 25 salient further on in the report. 13 1 Q. Thank you. To be clear, that's a theoretical or ideal 2 situation that you set out, ie the paradigms of design, 3 et cetera, rather than relating to this system? 4 A. That's right. This is all theory. It has nothing to do 5 with the actual documents I reviewed. 6 Q. In section 4 of your report you introduce the Horizon 7 System. You explain in summary terms what the system 8 is, how it was structured and how the system evolved 9 over time? 10 A. Yes. 11 Q. Just by way of summary, on page 9 of your report, the 12 paragraph at the top in the (a), (b) and (c) -- thank 13 you -- you detail in summary form the three major 14 iterations of the Horizon System; is this right? 15 A. That is correct. 16 Q. Firstly, the original system introduced into branches 17 from 1999 onwards and active until 2010, now known as 18 Legacy Horizon, although presumably not known as Legacy 19 Horizon at that time? 20 A. That's correct. 21 Q. Then the first major iteration of the Horizon Online 22 system, known as HNG-X, which was introduced in 2010 and 23 active until around 2017? 24 A. Yes. 25 Q. Then the second major iteration of the Horizon Online 14 1 system, introduced in about 2017 and still active today, 2 HNG-A, which I think is Horizon Anywhere? 3 A. Yes. 4 Q. The third thing you do, in section 5 of your report you 5 introduce Horizon's error logging and remediation 6 systems? 7 A. Yes. 8 Q. Then in section 6 you explain in more detail the 9 materials provided to you? 10 A. Yes. 11 Q. Thank you. 12 Can I turn to the limitations on your report. In 13 paragraphs 2.7.1 to 8, which is on pages 10 and 11 of 14 your report, you identify a series of limitations to 15 your report and therefore of the evidence that you can 16 give today and in November; is that right? 17 A. That is correct. 18 Q. I think summarising them, there are, I think, six or 19 seven of them: firstly, the documentation on which you 20 relied was a quarter of a century or so old, was written 21 for an internal market and not for the purposes of 22 subsequent forensic examination in legal proceedings; is 23 that right? 24 A. That is correct. 25 Q. Secondly, the documentation relates to, principally, the 15 1 period from '96 and 2000, reflecting the focus of your 2 report being on the rollout of Legacy Horizon? 3 A. Yes. 4 Q. Thirdly, given the nature, extent and duration over time 5 of the Horizon System, you could have spent an unlimited 6 amount of time researching and analysing it? 7 A. Indeed. That is correct. 8 Q. I think you're going to tell us in a moment that the 9 documentation related to Horizon -- that's training 10 manuals, operating instructions and the like, the 11 Horizon documentation -- itself amounts to over 100,000 12 documents? 13 A. That is correct. 14 Q. That's documents not pages? 15 A. Yes. 16 Q. Fourthly, it was in the nature of the task that you 17 undertook that you were focusing on material that tended 18 to describe problems and difficulties, rather than 19 trumpeting the accomplishments of Horizon? 20 A. Yes. 21 Q. Fifthly, is this right, given the technical nature of 22 the error logs, PinICLs, PEAKs and KELs, you may have 23 missed nuances or subtle shades of the use of language 24 within them, which nuances and shades may have been 25 evident to those responsible for actually using the 16 1 system? 2 A. Yes, that is correct. 3 Q. Sixthly, the PinICLs and PEAKs that you examined came 4 from the third line of Fujitsu's IT support and, 5 therefore, you didn't examine records relating to the 6 first and second lines of IT support? 7 A. That is correct. 8 Q. Lastly, as you've observed already, most of the material 9 you examined originated from Fujitsu and not the Post 10 Office and so you don't have any insight into Post 11 Office's views during the period which you're examining? 12 A. That is correct. 13 Q. Can we turn, then, to the first part of your report, 14 which is section 3 on page 13. 15 Can that be displayed, please. Can we highlight 16 3.1.1, please. Thank you. 17 You say, and I'm going to read it into the record, 18 that: 19 "To properly understand software systems, it is 20 important to appreciate how they fit into the overall 21 execution of the enterprise they support. Software 22 systems are enablers, not panaceas. In the best 23 situations, software applications can decisively improve 24 the execution of the enterprise's strategy by 25 streamlining operations. This often includes providing 17 1 complete and accurate reporting that informs decision 2 makers in a timely manner. In the worst situations, 3 mismatched expectations and/or faulty designs and 4 implementations degrade the execution of the 5 enterprise." 6 Can you explain what you were conveying in that 7 paragraph, please? 8 A. Certainly. I believe many people think that software 9 cures everything, that software is the leader of the 10 execution of an enterprise. What I'm trying to 11 emphasise here is that software is a tool that the 12 enterprise should be using in order to execute the 13 strategy and tactics that it has pre-defined, rather 14 than the other way around. The software does not define 15 the strategy and tactics; the software is a servant to 16 the strategy and tactics of an enterprise. 17 Q. You then set out, over the following paragraphs, the 18 five components that permit execution of the enterprise 19 and the first of those, the model components, is 20 strategy. Can you explain what you mean by "strategy", 21 please? 22 A. Strategy, as I say in my report, is the very high level 23 driver of what, you know -- what an enterprise is trying 24 to accomplish and the way it's trying to accomplish it. 25 This is often used, you know -- or often encapsulated in 18 1 mission statements and vision statements, and it is 2 a very -- usually a very straightforward, simple to 3 understand set of concepts that is the DNA, basically, 4 of an enterprise and what they're trying to do and, in 5 general, how they're trying to accomplish it. 6 Q. So this is in the form of a mission statement or 7 statement of purpose. It's focused on the organisation 8 and not the IT system? 9 A. That is correct. 10 Q. You set out at the foot, it's on the page now, the foot 11 of this page, the UK Post Office's Statement of Purpose 12 as at the time that you were writing your report. At 13 I think the top line of it is: 14 "We're here, in person, for the people who rely on 15 us." 16 It goes on to explain what it means by those three 17 component parts. 18 A. Yes. 19 Q. We needn't read those, but that is a Post Office 20 strategy? 21 A. Yes. 22 Q. You then explain -- over the page, please -- the tactics 23 or business operations of an enterprise and you say, in 24 paragraph 3.2.2, I'll read it in: 25 "To execute the strategy, it is important to have 19 1 a mature and well-understood set of policies and 2 procedures. Designing, developing and implementing the 3 tactical playbooks that control the day-to-day business 4 operations across all aspects of the enterprise takes 5 considerable effort. The balance between aspirational 6 goals and realistic constraints is the responsibility of 7 those put in charge of making 'real-world' decisions 8 that affect how an enterprise is operated." 9 Again, this: the tactics need not refer to the 10 IT system? 11 A. That's correct. 12 Q. It might do but it does not necessarily do so? 13 A. That's right. 14 Q. The tactics would obviously be guided by the strategy? 15 A. Yes. 16 Q. You explain, as a third component part, in 17 paragraph 3.2.3, about the concepts of software systems 18 and you say: 19 "A software system's sole purpose is to efficiently 20 reinforce the business operations." 21 A. That is correct. 22 Q. So the tactics select software systems based on their 23 ability to conform to the defined business operations 24 requirements of the tactics and the strategy; is that 25 right? 20 1 A. Yes. 2 Q. You then speak to the fourth component part, 3 paragraph 3.2.4, "Data Management (Facts)", what did you 4 mean by "facts"? 5 A. Oftentimes, information systems will be systems of 6 record. For example, if it's an accounting transaction, 7 it will record that transaction and I would consider 8 that particular set of data a fact for the enterprise. 9 Q. So data management is governed by the design 10 specifications of the software systems? 11 A. Yes. 12 Q. You say in the second sentence there: 13 "The management of these facts requires alignment of 14 the software systems to the business operations and 15 anticipates downstream analytics and reporting." 16 What did you mean by that second sentence, please? 17 A. So, oftentimes, facts are accumulated in a voluminous 18 nature and the -- one of the benefits of having 19 a software system collecting all of this information is 20 to further analyse and report on it. In order to do 21 that correctly, number 1, the software system has to 22 direct the collection of data in a structured and 23 understood manner so that the reporting and analytics 24 that can be performed on that is well defined and well 25 understood by everyone throughout the enterprise. 21 1 Q. You turn, fifthly, to analytics and reporting and this 2 part of the model represents how the enterprise 3 understands the data collected and managed through 4 a series of manipulations and summaries of the data 5 itself; is that right? 6 A. That is correct. 7 Q. Those rely on the rules employed by the data management 8 function? 9 A. Yes. 10 Q. You explain the hierarchical relationship between these 11 components in 3.4.2 of your report on page 15. You say 12 that the two concepts that should be considered that 13 affect a healthy, long-term relationship between the 14 components are adaptability and complexity. Under 15 "Adaptability", in (a)(i), you say: 16 "The downstream components should respond to the 17 requirements of the upstream components, not dictate 18 them ..." 19 Can you explain, please, what you mean by 20 "downstream" and "upstream" components? 21 A. Certainly. So the relationship or the hierarchy within 22 the model that we just went through has a clear pecking 23 order: strategy guides the tactics, tactics selects the 24 software, software controls the data management and the 25 data management supplies information to the reporting 22 1 and analytics section. 2 So that's the relationship that I'm referring to. 3 Q. Which of those are downstream and upstream? 4 A. So strategy is at the top of the hierarchy and reporting 5 and analytics is at the bottom of the hierarchy. 6 Q. What do you mean by "not dictate them"? Can you give 7 an example please? 8 A. So what I mean is there should never be an instance 9 where, let's say, the reporting and analytics defines 10 what the strategy should be. The reporting and 11 analytics should always be responsive to the strategy, 12 not dictating the strategy. That works the same up and 13 down the hierarchy that I've described. The software 14 system should never dictate the tactics of 15 an enterprise. The tactics of an enterprise should 16 always be in charge of the software system and what the 17 software does. 18 Q. You actually give an example in (ii) there. Could you 19 flesh that out a little bit, please? 20 A. Certainly. So in this example what I'm describing is 21 that if the reporting and analytics took it upon 22 themselves to expand on the information that was to be 23 collected, in theory, that should have been guided by 24 the tactics and then were sponsored by the software 25 system. It is possible that there are situations where 23 1 someone in the reporting and analytics division of this 2 hierarchy felt as though "Well, it would be very nice to 3 have this particular piece of information available but 4 it's not being collected by the software system, so 5 we're going to take it upon ourselves to start 6 collecting that information". 7 That, in the short run, could be a very good idea. 8 However, as that type of attitude towards the enterprise 9 goes on and on, you find that you're doing a lot of 10 one-off, ad hoc additions in the wrong place -- and in 11 this situation, I mean in the reporting and analytics -- 12 of collecting information, and not necessarily everyone 13 within the enterprise even knows that you're collecting 14 that information and they certainly aren't governing it. 15 There aren't any rules governing the collection of that 16 information, at least not from the purview of the 17 strategy or the tactics section. 18 So, in my experience, what I found is when that 19 happens it's almost as though a new kingdom has been set 20 up in the wrong area of the hierarchy and, over time, it 21 becomes disjointed with the strategy and tactics of the 22 enterprise and it creates an unstable situation because 23 once the strategy -- you know, once the senior leaders 24 or the line leaders of a system are aware that this 25 information is available and start relying on it, but 24 1 it's not really being controlled properly through the 2 software system or the data management component of the 3 process. 4 Oftentimes the integrity of that information is not 5 the best, and the situation then arises that no one 6 knows, really, who is in control of that information. 7 No one has a view onto the efficacy of the information, 8 and it is, essentially, out of control. The process -- 9 you've introduced a lack of control into the process, 10 and that has a lot of knock-on effects down the line, 11 and any one addition like that seems fairly innocuous 12 but, over time, it's as though bile is being collected 13 in the system and, eventually, things become very out of 14 control, if you are not adhering to who's in charge and 15 where the proper division of labour should be for that 16 particular example. 17 Q. I think you mention some species of that bile in the 18 last sentence of (ii): 19 "... inefficiencies of communication, maintenance 20 and costs." 21 A. That's right because, once you start going down that 22 path where you're out of control, as far as the 23 collection, you know, the proper collection and 24 maintenance of that data, then people are expecting -- 25 like, for instance, if this was done completely in the 25 1 reporting and analysis section, people further upstream, 2 such as in the tactics or in the strategy section might 3 assume that it's being -- that it's part of the software 4 system or part of the data management system, where it's 5 not. And the data -- 6 You know, it's extremely inefficient to have 7 multiple people doing the same thing and what I'm trying 8 to explain here is that, as these lines are blurred, no 9 one knows really who has responsibility over the data, 10 and no one knows where the data is coming from, which is 11 the communications issue, and as systems are upgraded. 12 So let's say that we did collect a whole bunch of 13 extra data in the reporting and analysis section and 14 then went through a change management -- a change 15 process in the software and the data management section, 16 they have no idea that this other information is being 17 collected and they don't know how -- they wouldn't know 18 how a change in the software or a change in the policies 19 and procedures that are in the tactics section, or even 20 a change in the data management portion of the hierarchy 21 are affected by the expectation that this extra data is 22 always being collected because they, in fact, might not 23 even know that this data is being collected. 24 Q. The second concept to which you refer, to ensure 25 a healthy long-term relationship between the component 26 1 parts, is complexity. You say in paragraph (b)(i): 2 "Current efficiency and future flexibility benefit 3 from complexity being localised as far as downstream as 4 possible." 5 Can you explain, please, what you meant by that? 6 A. Certainly. So the systems are complex. You know, 7 running an enterprise is complex. But it's important 8 for guiding principles at the top to then -- as it 9 filters its way down through the hierarchy, to then be 10 more real world. The concept of pushing complexity down 11 as far as possible allows rapid changes to be made to 12 the system, whereas -- opposed to if all of the complex 13 existed at the top of the hierarchy, it would require 14 a vast amount of changes downstream because every change 15 at the top cascades down. 16 So, to the extent that you can push down the 17 complexity as far as possible, it limits the amount of 18 adjustments that need to be done as things are changing, 19 you know, on the different parts of the hierarchy. 20 Q. Thank you. Again, in (ii), you give an example of not 21 adopting that philosophy. Can you remind yourself of 22 that example and then try to explain it to us, please? 23 A. Right. In this example what I'm -- the example I'm 24 using is if a particular reporting requirement dictated 25 by the tactics section was inadvertently put into the 27 1 software selection section. There, the example I'm 2 using is whether a particular postal code is related to 3 an offshore isle or not. It is possible to locate that 4 particular functionality within a software system, it 5 certainly is, but that is a redundant piece of 6 information, in my perspective. 7 So, for instance, you have a UK postal code and the 8 tactics section might require an extra entry of checking 9 off a box to say whether this postal code relates to 10 an offshore isle. You certainly can require your 11 software system to record that but the correct place to 12 record that information is more in the data management 13 system, because it is redundant, there is no need to 14 introduce that into a user interface screen. It's 15 defined. It's pre-defined. Everyone, you know -- that 16 is something that can be managed further down in the 17 hierarchy. 18 If you did require the software to do that, here are 19 the downfalls of that: so number 1, the extra amount of 20 coding that it would take just to implement that one 21 little change. But the bigger issue is you are now 22 allowing the possibility for, internally, your data to 23 not be correct. So you're giving the user the option of 24 giving a postal code and a particular category or tag to 25 that postal code. The user might put that in 28 1 incorrectly. However, if you have that pushed down into 2 the data management section and the data management 3 section has a definition of each postal code and whether 4 it is an offshore isle, it's taken that labour off the 5 user it's taken the labour out of creating the software 6 and it's also maintaining the integrity of the reporting 7 that will be using that. 8 Q. In your answer there, you gave, as one of the 9 consequences of not adopting this approach, the need for 10 writing much more additional code: "extra coding", 11 I think you called it. 12 A. Yes. 13 Q. Is there a problem by having to write additional code 14 and, if so, what is it? 15 A. Well, the first problem is that, if that code is not 16 necessary, there's going to be a cost component to that 17 code. So that's the first problem. But the second 18 problem, really, has more to do with being -- having 19 internal referential integrity of the data and that's 20 what I just described. It is possible, if you put that 21 in, to have data that doesn't agree with each other. If 22 I have a particular postal code that is related to 23 London and I have the option of checking that off as 24 being an offshore isle, I've just introduced a data 25 error into the system. 29 1 Then the third thing would be the maintenance of 2 that code, the maintenance of the software, to the 3 extent it's ever upgraded, would have to take this extra 4 coding into account and have a knock-on effect of 5 perhaps increasing the maintenance costs further down 6 the line. 7 Q. Where does the concept of data-driven logic -- something 8 that you're going to speak about in a moment, 9 I anticipate -- fit in with what you have just 10 described? 11 A. So data-driven logic would be what we just described as 12 the proper placement for this particular example. It is 13 a reference table -- in this example would be 14 a reference table for all the postal codes and, instead 15 of having hard-coded in as software, we could have 16 a reference table, basically, which just basically means 17 I have a list of all my UK postal codes and then I have 18 an indicator of whether that postal code should be 19 considered an offshore isle or not and the people 20 maintaining that particular database, you know, let's 21 say that there was -- you know, you had a new isle all 22 of a sudden pop up. It would be much easier to maintain 23 that reference in the data management section rather 24 than in the software part of the hierarchy. 25 Q. Can we turn to systems development, please. In 30 1 paragraph 3.5.1 of your report, you make a point as to 2 the distinction between the terms "software" and 3 "system"; could you explain that, please? 4 A. Certainly. Software I would describe as application 5 code. Application code is what controls all the 6 hardware but a system is a more universal charm, which 7 includes hardware and communications and a number of 8 other things. It's not simply just the software -- it's 9 not just the application code; it's the universal system 10 of all the components that are related. So, for 11 instance, you know, making sure that your communication 12 lines are working right; making sure that your printers 13 are working right; making sure that any other pieces of 14 hardware related to the system is included. 15 Q. So the system is how the software and the hardware 16 operate together? 17 A. Yes, that is correct. 18 Q. A subset of that is the software, and that's the system 19 or the part of the system most often known as 20 an application, that directs, in particular, the 21 computer's hardware? 22 A. Yes, that's correct. 23 Q. You explain in paragraph 3.5.5 the nature of hardware 24 devices. I wonder whether we could look at that, 25 please. I appreciate that, to you, a lot of this may be 31 1 very basic indeed, as indeed to a number of people 2 listening or watching online. But I want to take it at 3 this level right at the beginning of the Inquiry for 4 a reason, please. 5 You mention a series of hardware devices and you 6 categorise them as, firstly, input devices. Can you 7 give some examples of those, please? 8 A. Certainly. As I say in my report, keyboards, mice, 9 touchscreens, card readers and, in fact, even some 10 storage devices at times will act as an input device. 11 Q. Secondly, you categorise some processing devices. Can 12 you explain those and give some examples, please? 13 A. Certainly. The CPU of the computer, or the brain of the 14 computer, is the main processing device. 15 Q. Storage devices, thirdly. Can you explain those and 16 give some examples? 17 A. So hardwares, memory, like CD-ROMs, anything that 18 retains -- that persists information. 19 Q. Is it right that a storage device could be either 20 an output or an input device? 21 A. That's correct. 22 Q. Could you perhaps give us an example of that? 23 A. Certainly. So if you are working on a spreadsheet that 24 perhaps you saved yesterday, you -- as you pull up that 25 spreadsheet, it's referencing the hard drive to pull up 32 1 the information that's on the spreadsheet. So at that 2 point in time, your hard drive is considered an input 3 device, because that's where your application is 4 receiving information from. 5 Then, as you make changes to that spreadsheet and 6 are done for the day and save it, that same hard drive 7 is being saved to and, at that point, it turns into 8 an output device. 9 Q. You say in your report, where you indeed give that 10 example: 11 "Even in this very basic explanation, we can 12 foretell the bleeding of meanings." 13 What did you mean by that? 14 A. So what I mean by that is that, depending upon the 15 context of what we're talking about throughout the 16 course of any discussion about something as complex as 17 systems development and deployment and maintenance, you 18 really need to understand the particulars and the 19 details of the situation at hand to totally understand 20 the implications of what's going on. 21 Q. In paragraph 3.6 of your report, you give an overview of 22 the different types of software. These, as you explain, 23 sometimes interact with the hardware of a system, and 24 sometimes they interact with other software, and 25 sometimes they interact with the user of a system. You 33 1 set out the four main categories of software. The first 2 is an operating system or OS software. Can you, please, 3 explain that and give some examples? 4 A. Certainly. So some examples would just be, like, 5 Microsoft Windows or Linux or macOS, which is Apple's 6 operating system. Essentially, what this does is it 7 provides the interface between the hardware and 8 everything else that happens on the system. It's where 9 a device driver sits, it's basically how -- it's the 10 rulebook for how the hardware of that particular system 11 is going to interact with any other bit of software. 12 Q. You say in your report the operating system software is 13 the low-level software that allows the software to 14 interact with the computer's hardware. What do you mean 15 by "low-level software"? 16 A. It's the baseline software that basically is the train 17 conductor for everything that happens on the computer 18 and what I mean by that is it allows everything to 19 interact with the hardware because, ultimately, you 20 know, a computer is a piece of hardware and there could 21 be multiple different pieces of hardware on that 22 computer. The operating system is the level of 23 instructions that allow the hardware to interact with 24 anything else that is on that particular computer. 25 Q. The next species of software that you describe is 34 1 a database management system or DBMS. Can you please 2 explain that and give some examples of it? 3 A. Certainly. So, oftentimes, it's needed to -- systems 4 need to collect and organise information in a structured 5 manner and the database management system software helps 6 to do this. Oftentimes it's in a structure but it 7 doesn't always have to be -- when I say "structured", 8 I'm really referring to like tabular formats. You can 9 often think of a database management system as being 10 a series of tables that hold information and can be -- 11 and have relationships to other tables. 12 So, like, an example would be perhaps I have a sales 13 system and that sales system I want to know who all my 14 customers are. So there might be a table that holds 15 just customer information and a reference key for that 16 customer information. But it might also have 17 a different table that keeps track of all the sales I've 18 made to that customer. 19 So what a database management system does is it 20 tries to organise that information in a way that 21 minimises the amount of space it takes to record all 22 that information and allows me to do some analyses on 23 that information. 24 Q. You give examples as Microsoft's SQL Server and the 25 Oracle Database. Can you explain those, what they do? 35 1 A. Certainly. They do exactly what I just described. So 2 Microsoft SQL Server and Oracle Database are both 3 examples of relational database systems, that would be 4 the more tabular structure form and they really underpin 5 most large, like, accounting systems and ERP systems. 6 Q. What's an ERP system? 7 A. Enterprise relationship platform system that -- it's the 8 general software that helps run an enterprise, so it 9 usually includes your general ledger as well as any 10 other accounting subsystems, like accounts payable, 11 accounts receivable, you know, your inventory system. 12 You know, any -- like an SAP would be an example of 13 an ERP system. 14 Q. What's an SAP? 15 A. SAP is a brand named ERP system that basically helps run 16 your enterprise, so it will do everything from financial 17 to operational services for your enterprise and, in 18 theory, is integrated so it allows all of those 19 different systems to speak to each other. 20 Q. The third species that you describe is application 21 software. Can you please explain what you mean by 22 application software and then give some real world 23 examples? 24 A. Certainly. So an application -- that is a very general 25 term. You know, if you asked me to sit down and create 36 1 an address book for you, to keep your calendar and keep 2 your contacts, if I programmed it for you, I would 3 consider that a piece of application software. The SAP 4 system that we just talked about, I would consider that 5 a piece of application software. Usually, it is a piece 6 of software that is built for a specific business, or 7 maybe even non-business, purpose but it usually is 8 custom built for a particular purpose, even things like 9 Microsoft Word or Microsoft Excel, I would consider 10 those pieces of application software. 11 Q. Even though they are not built for a specific business? 12 A. That's right, but they are built for a specific purpose. 13 Q. Lastly, fourthly, you describe the fourth species: 14 application development software. Can you please 15 explain what application development software is and 16 give us some examples? 17 A. Certainly. So if you were to ask me to build you, you 18 know, a contact tracking system, perhaps I might want to 19 use what I'm referring to here as an application 20 development software system and what that is is it is 21 a set of software packages that allow programmers to 22 efficiently design, develop, deploy and maintain 23 software. So it's specific to systems development, 24 design and deployment, and it supports those -- you 25 know, that effort in organising all the code, organising 37 1 the releases and keeping track of that. I believe that 2 I -- you know, like Microsoft -- Microsoft has a studio 3 it's called Visual Studio, it is an application 4 development software, and Android also has a studio, if 5 you wanted to -- 6 So if I wanted to deploy a mobile app on Android, 7 I could use Android Studio's application development 8 software package to help me do that. 9 Q. So it's software for writing -- 10 A. Yes, it's software for writing software. 11 Q. -- and maintaining and amending and changing software? 12 A. Exactly. 13 Q. You express a caveat at 3.6.2 that there are many other 14 types of software but those four categories allow you, 15 in this report, to illustrate how software types 16 interact with each other? 17 A. Yes, that's correct. 18 Q. You give an example at 3.6.3 and can you just explain 19 that to us, please? 20 A. Certainly. In this example I'm talking about if we are 21 developing an accounting application, the first thing 22 that we would use as the developers of the accounting 23 application would be the application development 24 software to that. Knowing that accounting uses a lot 25 of -- or expects a lot of transactional information, 38 1 I would also expect that a database management system's 2 piece of software would be used to help record and 3 retain that information. Both of those would be 4 obviously, as I said before, interacting with the 5 operating system software and, as it was developed, all 6 of that would be considered an application. 7 Q. Thank you. In paragraph 3.7 and following of your 8 report, you explain to us the concept of the software 9 development life cycle or the systems development life 10 cycle, in both cases shortened to the acronym SDLC. 11 Could you explain the difference, if there is any 12 difference, between "software DLC" and "systems DLC"? 13 A. Certainly, much like we discussed a few moments ago, the 14 distinction is, if it was just software, I would be 15 concerned only about the application code here but, 16 taking a more universal view on the topic, and I want 17 to, as -- as a system is deployed, it is not simply just 18 a software. If I want to take into account things like 19 hardware and communications and all of the things 20 outside of the purview of this software, I'd want to 21 describe it as a systems development life cycle. 22 Q. You focus in your report on the latter of those: the 23 systems development life cycle; is that right? 24 A. That is correct. 25 Q. You explain that, although there are a variety of 39 1 approaches in practice across teams, there are seven 2 commonly used stages; is that right? 3 A. That is correct. 4 Q. The first of those is planning. Although that may be 5 obvious from the word, can you explain in this context 6 what is meant by it? 7 A. Yes. So planning, as I say in my report, this is the 8 stage that determines what's being requested and trying 9 to just put together an overarching plan of how you 10 would approach fulfilling that particular request. It 11 is very closely joined with the next section, which is 12 the analysis. So I would almost talk -- distinguish 13 these two as much like the strategy and the tactics of 14 a particular development of a system. 15 Q. You say that analysis, secondly, is the stage where the 16 design team gathers as much information as possible 17 about every detail of the requested system and covers 18 issues such as functionality, performance, equipment and 19 cost; is that right? 20 A. That is correct. 21 Q. Then the third stage, design: can you explain what's 22 involved in that stage, please? 23 A. Certainly. The design is basically the roadmap for how 24 you are going to achieve the goals set out in the 25 planning and analysis stage. This includes a lot of 40 1 different things. It is considering both the 2 architecture of just the software, as well as how the -- 3 what hardware is required and making sure that the 4 design of the software is properly accounting for the 5 required hardware that's associated with the system, 6 including communications, including all of the upstream 7 and downstream processes. 8 So, for instance, I might want to bifurcate or 9 trifurcate my design into here is what the user is going 10 to see, here is what the operational -- the operations 11 of the communications between perhaps a bunch of 12 satellite users and a central repository of information 13 that's going to be collecting all of that information 14 for users. We need to understand what is going to be -- 15 how this information is going to be consumed, what needs 16 to be done with it. It is trying to take a very 17 structured, rigorous approach to understanding not only 18 what is being requested right now but also perhaps 19 anticipating that changes might be required in the 20 future. So kind of baking that into the structure of 21 the way this system is designed right now to 22 accommodate, hopefully, you know, reasonably 23 anticipatable future requests. 24 Q. You say in this paragraph: 25 "If an external resource is determined to be 41 1 appropriate, an integration portion of the design will 2 be documented." 3 What did you mean by that? 4 A. Certainly. So oftentimes, especially on large projects 5 or complex projects, the team that is -- that has 6 assumed the role of the general contractor for 7 a particular piece of software or a system, I should 8 say, rather, might not need to develop every bit of 9 technical feature from scratch. They might be aware 10 that there are components that exist right now from 11 people outside of their particular programming staff 12 that -- that functionality already exists. 13 So, to the extent that they get to a buy or make 14 decision, they might decide that they would prefer to go 15 out there, out to the market and purchase an existing 16 piece of technology and incorporate that into the system 17 that they're developing. 18 If they do that, they need to be -- they need to be 19 well coordinated with that third party that is providing 20 a particular function or a particular feature that's 21 going to be incorporated into the system, so that 22 everyone knows exactly what's expected. Everyone knows, 23 you know, the -- because there's a lot of technical 24 details when you're incorporating someone else's piece 25 of software, someone else's solution for a particular 42 1 function of your system. 2 It's very important that everyone understand exactly 3 what's expected from both sides so that it operates 4 correctly when you actually fold everything together and 5 deliver what you're calling a system. 6 Q. Thank you. The fourth stage is development and you 7 explain that using the technical design document from 8 the previous stage, the development team will transform 9 the design into a functioning system. 10 A. Right. So this is where it goes from theoretical to 11 practical. This is the -- once the design document has 12 been created, it is then used as basically the recipe 13 book for the development team to actually code the 14 software, to do the integration of the hardware with the 15 software that will create the system and that will 16 include, you know, hardware such as, you know, printers 17 or touchscreens, but as well as making sure that things 18 like communications systems are working properly, so 19 that all the different components of the software or all 20 the components of the design, many of which are 21 software, but are connected by different hardware 22 pieces. 23 So the development is taking the design, which is 24 the theoretical -- the theoretical roadmap for the 25 system and actually turning it into a real piece of -- 43 1 a real system which includes all the hardware and 2 software components. 3 Q. To be clear, this is the stage at which code writing or 4 coding occurs? 5 A. That is correct. 6 Q. The fifth stage is testing. You say: 7 "This phase is used to ensure that the results of 8 the development phase align with the expected 9 functionality, performance and hardware described by the 10 technical design document." 11 Is this phase an important one, the testing phase? 12 A. Oh, yes, of course it is. The design provides the 13 roadmap. The development is the actual application of 14 that roadmap to make something -- to make a real piece 15 of -- a real system which includes the software. But we 16 need to make sure it works correctly and, in order to do 17 that, there is always a rigorous testing process that 18 accompanies the initial deployment of the software or of 19 the system. 20 Q. You explain that there are two levels of testing: 21 quality assurance, QA, and then user acceptance testing, 22 UAT, yes? 23 A. Yes, so oftentimes, or most of the time, the testing 24 first is done internally by the same group that is 25 writing the software and there's a division of labour 44 1 within that group. Usually, there are the developers -- 2 or there's the designers. But there's the developers, 3 and then there is a different group within that 4 particular firm that will test it. It's important that 5 they be independent of the development group for 6 multiple reasons but the most important one is they need 7 to have an independent view on whether the system that 8 was created by the development group actually adheres to 9 all of the design specs that came out of the design 10 group. 11 So you have an internal team that will go through 12 a battery of tests, it's usually a very rigorous set of 13 tests that make sure that everything that they see in 14 the actual development of the system adheres to the 15 design specifications that was given to the developers 16 but independently verified by the testing group, by what 17 I'm calling QA, quality assurance. 18 Q. You have sometimes spoken in the present tense there. 19 To what extent was that about which you just spoke 20 commonplace 20/25 years ago? 21 A. I would say that, as long as software has been 22 developed, in my experience, which has been since the 23 '90s, that a QA function has also existed. 24 Q. You emphasise that this group should be a separate group 25 of professionals but within the development and design 45 1 team? 2 A. Yes. Yes, it's not the developers but it is from the 3 same enterprise as the developers. 4 Q. Yes, the same company? 5 A. Yes. 6 Q. You say it should be independent. I think you emphasise 7 why that was. You say there's a range of reasons, 8 presumably not marking one's own homework is one reason? 9 A. Right. I mean, practically speaking, even when you're 10 writing a report, it's always good to get a fresh set of 11 eyes on the report to see things that perhaps you're 12 blind to. So that's just a practical aspect of having 13 an independent group of people do the same thing in the 14 context of software systems. It's just good to get 15 a fresh set of eyes on something. 16 It's also good to have an independent group because 17 the roles are different. The structure and rigour 18 around a group of programmers that do testing is 19 different than the structure and rigour of a group of 20 programmers that do development. 21 Q. The second species of testing or level of testing you 22 describe as: 23 "User Acceptance Testing ... A small group of users 24 from the group requesting the system then performs 'real 25 world' testing to make sure the system meets their 46 1 expectations." 2 Can you explain in a little more detail what's 3 involved? 4 A. Certainly. So once a system has gained approval by the 5 quality assurance group of testing, the first group of 6 testers, a company would have two options. We can 7 either roll this software out to the entire user 8 community or we can roll it out to a very small group of 9 users to make sure that it's acceptable to them. The 10 benefit of rolling out to a small group of users is to 11 identify operational issues, is this system 12 understandable to you, as well as to catch maybe some 13 errors that slip through the cracks of the quality 14 assurance. 15 The reason I said it was a benefit is, oftentimes, 16 the user community and the developer community are two 17 completely divorced communities. What the design and 18 development team might think of as a great way to 19 operationalise something in a system might not be as 20 appetising to actual users of that system and if you 21 roll it out to a small group of users in this user 22 acceptance testing, you get the opportunity to get more 23 stylistic feedback, as well as doing one extra level of 24 testing to make sure that the functioning of the 25 software or the system is performing as needed. 47 1 Q. You explain in this paragraph that often there are 2 certain benchmarks that define whether the system can be 3 permitted to go to the next stage, the deployment stage, 4 ie a written down, recorded set of criteria; is that 5 right? 6 A. That's correct. 7 Q. You explain that the system does not need to be perfect 8 to be deployed but it needs to be acceptable to the user 9 community? 10 A. That is correct. 11 Q. So one will often see criteria developed and the 12 performance and operability and functionality of the 13 system measured against those criteria? 14 A. That is correct. 15 Q. The next stage is deployment. Can you explain what 16 happens at that stage, please? 17 A. So once user acceptance testing has passed, has given 18 the system a passing mark, it's now time to take this 19 system and make it accessible to the entire anticipated 20 user community, and deployment is that process where you 21 are now rolling out the software to the entire 22 population of users anticipated, you know, through this 23 process -- 24 You know, when the software gets -- when the 25 agreement to make the software happens, you anticipate 48 1 what the entire user community is. The user acceptance 2 training -- testing was a small set of it. The 3 deployment is talking about now rolling it out to 4 everyone, making sure that -- or allowing everyone to 5 access this particular system. 6 Q. You explain that this can be done in stages or all at 7 once. 8 A. That is correct, depending on the circumstances. 9 Sometimes it is advisable to go ahead and release this 10 particular system to everyone all at once. Other times, 11 maybe there are logistical issues that make it more 12 advisable to roll this out to 10 per cent of the user 13 community this week, 10 per cent next week, 10 per cent 14 the week after. It just might be a logistical issue, 15 but both deployment strategies or both deployment 16 options are available, and it really just depends on 17 agreement between the people contracting for that system 18 and the people delivering the system. 19 Q. You say that this stage involves the delivery of 20 documentation to users concerning the operation of the 21 system? 22 A. Yes. So as the system is rolled out, you will then also 23 need to make sure that the proper support for the users 24 exists and that is done in two forms: usually a user 25 guide and access to a help facility, meaning either, you 49 1 know, a phone call to a helpdesk, an email, some sort of 2 communication mechanism, to the extent that users do 3 experience issues, that they have something besides the 4 documentation. They should refer to the documentation 5 first but, to the extent that that's not helping them in 6 their particular situation, they need to have access to 7 someone else that can help them in realtime. 8 Q. You describe this as a contract mechanism (sic) for the 9 system's helpdesk. What did you mean by that, 10 a "contract mechanism"? 11 A. A "contact mechanism"? 12 Q. Ah, "contact mechanism", I misread the word. You just 13 described it. 14 A. Yes -- 15 Q. Please ignore that question. 16 A. -- it's how you get in touch to the helpdesk to the 17 extent that you need the help. 18 Q. You included in the answer before last a mention of the 19 need for training as part of this stage. 20 A. Yes. So depending on how complex the system is, in 21 addition to the training manuals and the access to 22 a helpdesk, it could require training, especially if 23 this particular system represents kind of a paradigm 24 shift, you know, where you're moving a lot of people 25 from doing something that they used to do one way or 50 1 never did at all and are just not familiar with the 2 entire concept of what we're trying to achieve here, and 3 how the software is -- or the system is helping you 4 achieve that. 5 Training is another avenue to make sure that the 6 users are well situated to employ and utilise the 7 system. 8 Q. Then the last stage is maintenance of the system when 9 it's in use. 10 A. Yes. Yes. So maintenance -- so even once we've gotten 11 to the point where the system has gone through all the 12 testing, all the training is happening and the user 13 community is interacting with the system, there is 14 a possibility that the users have identified some bugs 15 or errors in the system, in which case those bugs and 16 errors need to be addressed. It also -- usually, when 17 a system is rolled out and to the extent that the user 18 community is excited about the system and sees the 19 potential of other things that the system can do, the 20 ability for the users to communicate those desires, for 21 new functionality, usually is collected during this 22 point. 23 The maintenance, therefore, is twofold: one, if 24 there are errors or there are bugs in the system, it's 25 to allow for the correction of the bugs. It's also to 51 1 act as a collection of basically wish lists of things 2 that the system could do in the future, to the extent 3 that everyone agrees that it's proper to go ahead and 4 create a different version of the system. 5 Q. Can we just complete this section of your report before 6 the morning break. In paragraph 3.8.1 you describe or 7 explain how: 8 "Over time, there has been an evolution of how the 9 stages of SDLC are modelled." 10 You describe, I think, the oldest model as being 11 a waterfall concept. Can you please explain what that 12 involved? 13 A. Yes. So in the past, a waterfall methodology was often 14 employed, a waterfall SDLC methodology was employed 15 which basically said I want to try to do everything in 16 a monolithic fashion. I want to know every design 17 aspect and get that set. I want to develop everything 18 in -- that is described in the design concept. 19 Basically, I want to do everything in each stage and not 20 move on to the next stage until the prior stage is 21 complete. So that's the old way of doing it. 22 In more recent times, what has happened is people or 23 development communities have broken up the design, 24 development and deployment into smaller chunks. So 25 they're not necessarily creating the entire system at 52 1 once but they're creating components of the system at 2 once and trying to move those components -- those 3 bitesize chunks through user acceptance and -- or 4 through design, development, testing and maintenance in 5 smaller chunks and that -- 6 What that does is it allows kind of a trickle effect 7 of getting the system out into the user community 8 a little faster, although be it (sic) in smaller 9 functional chunks than the entire system at once. 10 Q. You described that as Agile development? 11 A. Yes. 12 Q. Would something we've seen in the papers here called 13 either -- Inquiry papers, rather than the newspapers -- 14 rapid application development technique, be a form of 15 Agile development? 16 A. Yes. Yes. So there's lots of different flavours and 17 there's lots of different nuances but, essentially, what 18 I'm trying to describe here is that you can -- there are 19 many different approaches to doing systems development 20 life cycle and, oftentimes, they're really around how 21 quickly we want to get things through, what level of 22 acceptance is required, maybe in a rapid level of 23 acceptance. You don't need it to be as perfect as in 24 a waterfall level of acceptance. 25 That's really a stylistic and taste choice on both 53 1 the developer and as well as the user and that's just 2 something that is -- there's constant -- there's a much 3 more frequent feedback loop in the rapid development as 4 opposed to the waterfall method. 5 Q. I described it in opening as an approach to software 6 development that focuses more ongoing software projects 7 and user feedback and less on following a strict plan of 8 development and testing cycles. 9 A. Yes. 10 Q. Does that sound about right? 11 A. That does sound right. 12 MR BEER: Thank you. 13 On that happy note, can we break for the morning, 14 please, sir, if it suits you. Just coming up to 11.25. 15 Can we say 11.40 or 11.45, sir? 16 SIR WYN WILLIAMS: Well, Mr Cipione, you are answering very 17 many questions. How much of a break would you like? 18 I'm very happy to extend the break until 11.45, if that 19 suits you. 20 MR BEER: Yes, thank you very much. 11.45. Thank you. 21 So we'll break until 11.45, thank you. 22 (11.25 am) 23 (A short break) 24 (11.45 am) 25 MR BEER: Good morning, sir. Can you see and hear me? 54 1 SIR WYN WILLIAMS: Yes, I can. 2 MR BEER: Yes, and we can see and hear you. Thank you very 3 much. 4 Mr Cipione, can we turn to section 4 of your report, 5 which starts on page 21. In this section of your 6 report, you set out a summary of the Post Office and its 7 branches, a summary of the services available at Post 8 Office branches, a summary of the Horizon IT System, 9 looking first at the components of Legacy Horizon, which 10 you describe as components (a) to (d), then you look at 11 the components of Horizon Online, again describing them 12 as components (a) to (d), and then you deal with the 13 important activities or the important concepts of 14 remming in and rolling over. 15 In order to provide those summaries, is it right 16 that you have drawn on the documents set out in 17 paragraph 4.1.3 of your report -- which I would ask to 18 be displayed, so if you just scroll down, thank you -- 19 in (a) to (f)? 20 A. Yes. 21 Q. So those six documents that are listed there are the 22 essential bases for what you say by way of summary? 23 A. Yes. 24 Q. You enter a caveat at the foot of the page at 25 paragraph 4.1.4, in which you say: 55 1 "I have endeavoured to summarise these documents to 2 what I consider an appropriate level of detail Inquiry, 3 but this has necessarily required me to omit some of the 4 extensive technical details ..." 5 You explain that one document runs through 819 pages 6 and another document runs to 417 pages. 7 A. Yes. 8 Q. So you have summarised but hopefully not oversimplified? 9 A. That was the intent. 10 Q. Can we start then with the Post Office and its branches, 11 turn to paragraph 4.2, please. You explain that 12 although the formal company name and structure of the 13 Post Office has changed several times over the course of 14 the last few decades, it's remained, in essence, 15 a government-owned company responsible for operating 16 a network of branches throughout the United Kingdom in 17 which it offers post and other services to the general 18 public? 19 A. Yes. 20 Q. Between 1986 and 2001 the part with which we are most 21 concerned was called Post Office Counters Limited, or 22 POCL, as you describe them? 23 A. Yes. 24 Q. From 2001, it was known as Post Office Limited? 25 A. Yes. 56 1 Q. You explain in your paragraph 4.2.3 the three different 2 species of Post Office branches. Firstly, Crown Office 3 branches and you explain that these are -- these 4 branches are directly managed by Post Office Counters 5 Limited and are known as "Crown" post offices. They're 6 run by employees of Post Office Counters Limited and 7 such employees are commonly known as Crown Office 8 employees? 9 A. Yes. 10 Q. The second species are agency post offices and can you 11 explain what you understood agency post office branches 12 to be? 13 A. My understanding is that these are branches that are 14 located in shops or other facilities around the UK and 15 are where Post Office services can be offered by the 16 shopkeepers. 17 Q. The distinction is that the branches were owned by the 18 subpostmasters who were agents of Post Office Counters 19 Limited? 20 A. Yes. 21 Q. The third species are outreach services and you describe 22 these as typically being small, part-time branches that 23 may use a village hall or mobile van to provide Post 24 Office services to communities that might not otherwise 25 receive them? 57 1 A. Yes. 2 Q. In a graph, if we can go over the page, please, which is 3 your figure 4.1, and if we could enlarge just the graph, 4 please. Thank you. 5 We can see the changing nature of those three 6 species of branches depicted in this figure 4.1. 7 I think this describes how many thousands of each type 8 of branch there were for the period 2000 to 2021? 9 A. Yes. 10 Q. I think, would this be right: the data shows firstly 11 a decline in the overall number from about 18,000-odd to 12 less than 12,000-odd? 13 A. That is correct. 14 Q. It would show, secondly, a decline in the number of 15 Crown Office branches, that's the purple on the graph? 16 A. Yes. 17 Q. I think you make the point in your report -- it's 18 paragraph 4.2.5, no need to look it up at the moment -- 19 that although, certainly in 2003, the Crown Office 20 branches represented only 3 per cent of the overall 21 estate, the Post Office said that they accounted for 22 over 20 per cent of transactions by volume? 23 A. That is correct. 24 Q. I think the third thing we can probably take from this 25 graph is that the number of outreach services that were 58 1 offered grew very substantially from 2000 up until 2021, 2 there depicted by the dark green on this graph? 3 A. Yes. 4 Q. In paragraph 4.3 of your report you explain the services 5 available at Post Office branches and you say at one 6 time it was estimated that some 170 services were 7 offered, and they include the well-known services listed 8 in your seven paragraphs (a) to (g), and these are all 9 examples of what you describe as transactions. What do 10 you mean by the phrase "transactions"? 11 A. In the context of the Horizon System, as each one of 12 these services were engaged upon by the customers 13 through the Horizon System, they would generate 14 a transaction that would need to be recorded within the 15 Horizon System. 16 Q. So, essentially, a transaction in this context is any 17 event in which a customer used a Post Office service in 18 a branch that needed to be recorded in a system? 19 A. That is correct. 20 Q. You make the point later in your report, it's 21 paragraph 4.3.6 -- no need to turn it up on the 22 screen -- that not all transactions were internal to 23 Post Office Counters Limited; is that right? 24 A. That is correct. 25 Q. Is that because Post Office Counters Limited was 59 1 providing services to clients, some in the public sector 2 and some in the private sector? 3 A. Yes. 4 Q. Can you give some examples of services provided to 5 public sector clients? 6 A. In 4.3.6, I describe the Driver and Vehicle Licensing 7 Agency and the Department of Work and Pensions would 8 have been public sector clients. 9 Q. Private sector clients, can you give some examples of 10 those, please? 11 A. Camelot, British Telecom would be examples. 12 Q. I think you mentioned Girobank too? 13 A. Yes. 14 Q. That meant that some of the money that was collected in 15 branch would need to be sent to, or indeed obtained 16 from, such clients but that was done by Post Office 17 Counters Limited; is that right? 18 A. That is correct. 19 Q. You make the point in paragraph 4.3.7 that it was 20 important to keep a record in the branch of all such 21 transactions so that Post Office Counters Limited could 22 work out which clients it needed to pay money to or 23 claim money from, as well as ensuring that its own cash 24 and stock was accounted for; is that right? 25 A. That is correct. 60 1 Q. You explain in paragraph 4.3.8 that before Horizon was 2 introduced, a number of branches would record their 3 transactions in paper form in ledgers or other similar 4 documents, or use their own electronic point of sale or 5 EPOS systems, one of which was called ECCO+? 6 A. Yes. 7 Q. The ECCO+ system, is that essentially the brand name or 8 product name of the supplier of that system? 9 A. That is my understanding, yes. 10 Q. When we mention transactions in this context, they do 11 not include occasions, is this right, where a customer 12 purchases an item in a shop that is co-located with the 13 Post Office, like confectionary or bread and milk or 14 a newspaper? 15 A. That's correct. The Post Office transactions or the 16 POCL transactions were taken care of on the Horizon 17 counter. All of the shop transactions were taken care 18 on -- through a different method. 19 Q. So the transactions that I mentioned, or the type that 20 I've just mentioned, would be processed separately from 21 those of the Post Office branch, often via a separate 22 counter? 23 A. Yes. 24 Q. So perhaps a number of us have experienced it. If you 25 wanted to buy a book of stamps and a newspaper, you've 61 1 got to get in two queues sequentially. 2 A. Yes. 3 Q. As we've seen from your table, the majority of Post 4 Office branches were agency branches. They were owned 5 and managed by subpostmasters -- 6 A. Yes. 7 Q. -- and the cash and the stock was owned by Post Office 8 Counters Limited but managed day-to-day by the 9 subpostmasters? 10 A. Yes. 11 Q. Can we turn, please, to the Horizon System and turn up 12 paragraph 4.4.1. Thank you. 13 You explain that the system was introduced in 14 stages, known sometimes as a rollout, between 1999 and 15 2000, and that the objective, as you understand it, of 16 the Horizon IT System implementation, was to modernise 17 the point of sale and managerial accounting functions 18 across the network of Post Office branches. Today we 19 might describe this process as "digitising" the branch 20 network? 21 A. Yes. 22 Q. You explain that the Horizon System is still in use 23 today, albeit it's gone through the three main 24 iterations that we have previously discussed in its 25 22-year or so lifetime? 62 1 A. Yes. 2 Q. Can we begin with the Horizon System and we're going to 3 call it Legacy Horizon, as it came to be known. Turn 4 over the page to paragraph 4.5 and the table at 4.1. 5 Thank you. You kindly set out a brief history of Legacy 6 Horizon in this table at 4.1. Can we just run through 7 it, please, so that we've got the larger milestones in 8 mind at this early stage of the Inquiry, please. Again, 9 this is extracted from the document set that you 10 mentioned earlier on; is that right? 11 A. Yes, that is correct. 12 Q. So if you can start, please, using this table to narrate 13 these ten or so developments in the history of Legacy 14 Horizon? 15 A. Certainly. So, as you can see the first entry, May '96, 16 the DSS and POCL jointly awarded the contract for -- to 17 ICL Pathway for what we're calling Horizon, although you 18 can see that there are a number of different variations 19 of that name in here; Pathway Project, Pathway Horizon 20 and so on. It was -- ICL Pathway at the time was 21 a wholly-owned subsidiary of ICL. Fujitsu acquired 22 80 per cent of ICL's shares in 1990 and purchased the 23 remainder in 1998, and ICL was fully integrated into 24 Fujitsu in 2002 and was renamed Fujitsu Services 25 Limited. 63 1 Q. Just before moving on there, this contract, the May '96 2 one, was a contract to develop an IT system that would 3 firstly replace the existing paper-based method of 4 paying Social Security benefits and, secondly, automate 5 the entire national network of post offices; is that 6 right? 7 A. Indeed. 8 Q. Yes. Can you move on to September 1996, please. 9 A. Sure. In September '96 was the Initial Go Live that was 10 implemented in ten Post Office branches and this was 11 an interim -- a system for Child Benefit payments and 12 was limited to that functionality. 13 In November '97 that system was extended to 200 Post 14 Office branches, still just remaining -- the 15 functionality just being the Child Benefit payments and 16 it was noted in my documentation that the deadline for 17 completion of the operational live trial of the IT 18 system, was missed by ICLPL. 19 Q. That's, at that time ICL Pathway Limited? 20 A. Yes. 21 Q. Thank you. 22 A. In March '98, an interdepartmental working group was 23 established to review the viability of the Pathway 24 Project and the consequences of cancellation. The 25 working group comprised officials from Treasury, Cabinet 64 1 Office, Department of Trade and Industry and the DSS. 2 In July 1998, the interdepartmental working group 3 reported that the Pathway Project remained feasible but 4 required successful renegotiation of the contract with 5 ICLPL. 6 In October 1998, attempts to renegotiate the terms 7 of the contract between DSS, POCL and ICL failed. In 8 May 1999 the original PFI contracted awarded to ICLPL by 9 DSS and POCL was terminated. DTI announced a new 10 partnership agreement between POCL and ICLPL. 11 In July 1999 POCL and ICLPL agreed a fixed payment 12 contract to automate the national network of post 13 offices and, in late 1999, the rollout of Horizon 14 occurred or commenced. 15 Q. You mentioned earlier in your evidence this morning one 16 of the two stages of testing or levels of testing was 17 UAT, user acceptance testing. 18 A. Yes. 19 Q. So far as you know, would that refer to the stages on 20 this table of September '96 and November '97? 21 A. Yes, that would be the user acceptance testing, you're 22 correct. 23 Q. Thank you. We can take that table down, please. Can we 24 turn to the functionality of Legacy Horizon. We are 25 moving to paragraph 4.5.2 of your report. You explain 65 1 that there are essentially two elements to it. The 2 first of which is the electronic point of sale or the 3 EPOS element. Can you explain what the purpose and the 4 function of the EPOS was? 5 A. The purpose and function of the electronic point of sale 6 component of Horizon was simply to capture the 7 transactions that occurred at the branches throughout 8 the network. 9 Q. So it included the purchases of Post Office products, 10 such as stamps and stationery, made by customers in 11 branch; is that right? 12 A. That's correct. 13 Q. Also transactions carried out in branch for the purposes 14 of products or the use of services provided by clients 15 of the Post Office, and the clients here are the things 16 you've mentioned already -- or the organisations you've 17 mentioned already: some public sector clients, DVLA, 18 DWP; some private sector clients, banks or Camelot. 19 A. That's correct. 20 Q. You explain, secondly, that the purpose and function of 21 the Horizon IT System was one of management accounting. 22 Can you explain what that is, please? 23 A. Certainly. So the transactions that were collected at 24 each one of the branches for -- throughout the network 25 needed to be consolidated and organised for purposes of 66 1 doing all of the managerial accounting. What I mean by 2 managerial accounting is I would consider the 3 transactions operational -- details of the operations of 4 POCL's agents, as well as their Crown Offices. 5 All of those transactions needed to be organised in 6 order for POCL to do their own internal accounting as 7 well as exchange information with all of their clients, 8 so the managerial accounting was a step in that process 9 to collect all of the transactions and manage them in 10 order to supply further processes that needed to be done 11 for their own internal financial accounting as well as 12 to exchange information with all of their client 13 partners. 14 Q. Thank you. 15 You explain in paragraph 4.5.3 that, in terms of the 16 size and scale of the data process and the code written, 17 both were substantial. 18 A. Yes, they were. 19 Q. You tell us that in 2003, the Post Office stated that 20 Horizon processed nearly 2 billion transactions 21 per annum? 22 A. Yes. 23 Q. But despite that, you say that it was a relatively 24 simple task, computationally? 25 A. Yes, each individual transaction or -- there weren't any 67 1 complex calculations associated with any of these 2 transactions but there were a vast number of 3 transactions. 4 Q. You say that it's no more complex than systems operated 5 by, for example, banks? 6 A. That is correct. 7 Q. You refer to an estimate that Legacy Horizon had over 8 3.5 million lines of programming code. What's the 9 general approach that a system designer ought to take to 10 writing code, in terms of its volume? 11 A. It's all -- less is always better, certainly. However, 12 the requirements for different systems required 13 different volumes of software code. But less is 14 generally a better rule than more. 15 Q. Why is less better than more? 16 A. Maintenance. Well, number 1, simplicity of the coding 17 aligns with a good structure of code. But, just as 18 importantly, to the extent that maintenance needs to be 19 done on the code, the less code that exists to begin 20 with, the less code there is to maintain as updates are 21 made to the code. It's just simpler: the smaller number 22 of lines of code, the easier it is to maintain. 23 Q. Is it possible to say whether this is a high number or 24 a low number or an average number of lines of code, or 25 can one not apply such descriptors to it? 68 1 A. On the face of it, this looks like a very large amount 2 of code. However, I have not looked at the code. 3 I don't know exactly what this code represents. So 4 I don't have an opinion whether this is an appropriate 5 amount of code or not. 6 Q. Just give us a comparison. The systems that you 7 mentioned earlier that you designed for General Motors 8 and WorldCom, how many lines of code would we be talking 9 about there? 10 A. I would say 20,000 lines or less of code. 11 Q. But, as you said, the number may be an indication that 12 it was written as economically as could be but is 13 a reflection of the number of tasks that needed to be 14 performed, or it's an indication that the code was not 15 well written. But you haven't subjected the code to 16 forensic analysis -- 17 A. That is correct. 18 Q. -- because that wasn't within your instructions? 19 A. That's right. 20 Q. You tell us that the documentation ran to more than 21 100,000 pages. What do you mean by "the documentation"? 22 A. So documentation around the Horizon System, there was 23 a lot of it. So documentation could be user 24 documentation, it could be updates to user 25 documentation, it could be technical documentation, 69 1 updates, it could be business processes; all of those 2 are encompassed in this count. 3 Q. This may sound a silly question but is that a high 4 number? Does it appear to be a high number? 5 A. It appears to be a high number to me. We need to take 6 into account versioning, though. I'm positive that this 7 probably encompasses, you know, version 1, 8 version 1.1.1, you know, of all of the different 9 dimensions of documents. So it does appear to be a high 10 number but I have not catalogued it or made 11 a determination whether it is excessive or not. 12 Q. You make the point that the system was created 13 specifically for the purposes of servicing the Post 14 Office branches and didn't have the added burden of 15 integrating existing technologies. 16 A. That's correct. 17 Q. Would that be a limitation on the possibility of 18 additional complexity of a system? 19 A. It would indicate that the complexity of the system was 20 completely defined by this process and not aggravated by 21 any environmental factors of an existing system. 22 Q. In paragraph 4.5.5 you say that the project was 23 "ambitious" in both "scale and scope" and you draw some 24 contrasts with the state of information technology "at 25 this time", ie from about '96 to about 2000. 70 1 You remind us that the -- or remind some of us that 2 the Nokia 3210 was the best-selling phone of 1999 -- 3 some of us would wish that that technology still 4 existed -- but it had a monochrome screen; is that 5 right? 6 A. Yes. 7 Q. It didn't have any touchscreen navigation; we had to 8 wait until 2007, I think, for that -- 9 A. Yes. 10 Q. -- and one couldn't access the Internet through 11 a browser on the phone? 12 A. That is correct. 13 Q. You tell us that at this time, only about a third of 14 people were estimated to have a personal computer -- 15 A. Yes. 16 Q. -- and only 30 per cent of adults had access to the 17 Internet? 18 A. Yes. 19 Q. We had to wait until 2004 for all of the benefits of 20 Facebook? 21 A. Facebook arrived in 2004. 22 Q. At this time, the IT world was focused on the so-called 23 Millennium Bug? 24 A. Yes. 25 Q. In terms of IT development, you tell us again here that 71 1 the prevailing method was the waterfall method, and 2 Agile development wasn't mainstream in IT development at 3 this time? 4 A. That is correct. 5 Q. Can we turn with that background to the seven elements 6 or aspects of the development and implementation of the 7 Horizon System, which drove its complexity. There set 8 out in your paragraph 4.5.6, which is at the foot of 9 page 28 and on to 29, and there are seven elements set 10 out in (a) to (g), can you talk us through those? 11 Firstly, the need to design a system that connected all 12 Post Office branches to a central server but could also 13 withstand a loss of connectivity? 14 A. Yes, so this was -- you know, at that time, this was 15 a much more difficult problem than it is now. Simply 16 because our communications infrastructure is much better 17 now. It's much more robust. The reliability of 18 connectivity, including the expense related to that 19 connectivity at this point in time, really provided 20 issues to anyone trying to maintain what I would refer 21 to as a client-server type process, meaning you have 22 satellite systems which were the clients and you would 23 have the central system which was a server. 24 This is just more a talking term. I'm not positive 25 I would describe the Horizon System as a client-server 72 1 but it's a good set of words to use in describing it. 2 The fact that they had to contemplate -- they being 3 ICL Pathway -- had to contemplate an extended loss of 4 connectivity meant that they had to put in guard grills 5 and safety nets for those circumstances where the -- 6 where they knew the connectivity wouldn't exist. So 7 they needed to not only create a design that allowed for 8 a system that is connected to work but they also needed 9 to design -- they needed to anticipate the fact that 10 they could not be connected. 11 So those were two different logistical issues that 12 they had to incorporate in their design and development 13 of the system. 14 Q. The point you're making here is that the system needed 15 to be designed so that it could maintain its 16 functionality or most of its functionality whilst there 17 was a lot of connectivity, i.e. customers could still be 18 served in the branch. 19 A. That's right, because it is -- the customer at each one 20 of the branches did not want a connectivity issue -- I'm 21 sure didn't want a connectivity issue to stop them from 22 purchasing stamps, for instance. So that required the 23 need for the design to anticipate connectivity issues. 24 Q. And then allow for correct synchronisation once 25 connectivity had been restored? 73 1 A. Right. So that's, you know, that's -- that complicates 2 the design and development of the system. So when 3 you're anticipating a loss of connectivity, you have to 4 have plan B. Okay, what does the system do now that 5 I know I'm not connected? Now I need to keep 6 a persistent store of the -- you know, I need to, 7 number 1, identify that I'm not connected and, number 2, 8 then I need to collect information until I know I'm 9 connected again and then, number 3, when I am connected 10 again, I need to make sure that the information that's 11 been stored up gets transmitted correctly to the central 12 server. 13 So those are -- that might sound simple, that is not 14 a simple process, necessarily. 15 Q. The second area of complexity you mentioned is the need 16 to integrate a variety of software and you mention, in 17 particular, Riposte and Tivoli. Can you explain what 18 riposte and Tivoli were? 19 A. Certainly. As I mentioned earlier, oftentimes in the 20 design of a system, you would decide whether to buy or 21 make certain functions within your system. In these 22 instances this a buy: I want to buy. So the Riposte was 23 a software that basically allowed for the look and feel 24 of the counter to be pre-made, you know, the 25 touchscreens, and all that, that was a product that was 74 1 already -- 2 Q. The user interface? 3 A. The user interface, yes. So Riposte provided that. It 4 also provided the mechanism for capturing and 5 transmitting the transactions that were related to all 6 the activity that happened on the user interface through 7 the counter. Tivoli was more of a behind the scenes 8 type product. It was more of an operational type 9 product but it assisted the system to update software 10 packages and update reference data, which we'll talk 11 about further on in my report but it was more of 12 an operational assistant to help the Horizon System work 13 properly. 14 Q. You also say that there was a need to integrate 15 a variety of hardware, including touchscreens, printers, 16 communications equipment, barcode scanners, weighing 17 scales, PIN pads, and the like? 18 A. Yes. There was a particular set up that was in the 19 design spec and those are the hardware components that 20 were aligned with that set-up. 21 Q. The third area of complexity that you mentioned is the 22 need to accommodate hardware failures because hardware 23 components in the 1990s were not as reliable as they are 24 today? 25 A. That is correct. 75 1 Q. The fourth element you mention is a large and diverse 2 user base amongst subpostmasters and the staff that they 3 employed, which would have included varying levels of 4 comfort using 'modern' IT systems, in inverted commas; 5 is that right? 6 A. That is correct. 7 Q. So you've got a cohort of people that are more or less 8 familiar and more or less happy with information 9 technology at the point of rollout? 10 A. Yes. 11 Q. You kindly note that Fujitsu itself noted that training 12 was provided to 63,000 staff from the ages of 16 to 13 87 years of age with various skills involved, and you 14 say that would, you believe, have presented 15 a significant training rollout and support challenge? 16 A. Yes. 17 Q. The fifth area of complexity you mention, I think, is 18 the volume of the rollout and you say that, between 19 August '99 and December 2000, over 14,000 branches had 20 Legacy Horizon installed? 21 A. Yes. 22 Q. You subsequently, in your report, set out in the table 23 at 4.2 -- no need to turn it up -- the progression of 24 that rollout, month by month, between August '99 and 25 December 2000? 76 1 A. Yes. 2 Q. The sixth area of complexity you mention was the 3 physical challenges of installing bulky IT hardware into 4 branches. Can you expand on that a little bit, please? 5 A. Yes, so the -- there was a hardware specification that 6 went along with the Horizon System which included the 7 counter printers, tape rollers, card readers and 8 whatnot. The branches might not have had space for 9 those and that presented logistical issues for -- 10 I mean, just the physical logistical issue to implement 11 the Horizon System at a branch. If they didn't have 12 space, they had that issue. 13 Additionally, there were communications constraints 14 at some of the branches. Some of them didn't have 15 access to some of the communication systems that the 16 Horizon System was designed for. 17 Q. So some of them didn't have an ISDN line; is that right? 18 A. That's correct. 19 Q. So they had to use a satellite link? 20 A. Yes. 21 Q. Lastly, seventhly, you mention a complexity that was 22 added because of the need for the system to be very 23 secure because, after all, it dealt with transfers of 24 money as well as containing personal information? 25 A. That is correct. 77 1 Q. You say, overall, that those challenges, in your view, 2 made the design, build and rollout of Legacy Horizon 3 very ambitious? 4 A. Yes. 5 Q. Can we turn, then, to the high-level design of the 6 Horizon System. This is over the page at 7 paragraph 4.5.8. So bearing those points of complexity 8 in mind, can you explain to us the elements of the 9 high-level design of the Horizon System, starting with 10 the fact that it was a system that used data-driven 11 logic rather than dealing with prices in its source 12 code; is that right? 13 A. That is correct. 14 Q. Can you explain this concept to me, the public and the 15 Chair, using the example that you give of hammers 16 screwdrivers and pliers costing £5, £7 and £6, 17 respectively, that you have included in your report, 18 please? 19 A. Absolutely. This is a very simple example, certainly, 20 but hopefully the concept will resonate as you think 21 about the more complex features of the Horizon System. 22 But what I'm trying to juxtapose is, to the extent that 23 we wanted to process a transaction for a hammer, 24 screwdriver and pliers through two different paths, one 25 path being source code path and one path being a source 78 1 code supported by reference data path. 2 In 4.5.12, what I'm attempting to do is to show what 3 source code might look like if it was the only arbiter 4 of processing this data. There would have to be -- 5 Q. So if you could walk us through the example that's 6 emboldened. 7 A. Certainly. I'll just go line by line and, if you have 8 questions, then you can ask them. 9 So the purpose of both of these sets of code is to 10 calculate a basket total for the purchase of three 11 items. So the first thing, the first function that 12 needs to happen is we need to set our total basket to 13 zero. We need to start at zero. Then we are going to 14 check if the product that's being -- if one of the 15 products that's being purchased is a hammer. If that is 16 correct, then I'm going to multiply the quantity of 17 hammers by £5 and add this to the total basket amount, 18 and you'll notice that this is what's referred to as 19 hard coding. 20 So this is hard coded software. So the -- no matter 21 how many hammers come through here, they're always going 22 to be multiplied by £5 if this source code remains the 23 same. 24 The next item I'm looking for is a screwdriver and, 25 if there are screwdrivers, I'm going to take the 79 1 quantity of screwdrivers and multiply them by £7 and add 2 that to the total basket amount. Then, finally, we're 3 going to look to see if the product is a pair of pliers. 4 If we do have a product being purchased as a pair of 5 pliers, we're going to multiply the quantity of pliers 6 by £6 and add that to the basket. 7 That will generate the total basket amount based off 8 this hard cod -- you'll notice at the bottom I also 9 checked to see if there are any products that are not 10 hard, you know, a hammer, screwdriver or pliers. That's 11 just a general error check that is commonly used in 12 code. 13 But the purpose of this is just to multiply the 14 number of hammers, pliers or screwdrivers by their 15 respective costs or purchase amounts. 16 Q. You've written this out, I think, in pseudo-code, not 17 the actual code that would be used. Is pseudo-code 18 a plain language description of the steps that might be 19 taken in an algorithm or another type of -- 20 A. This is supposed to be a plain language representation 21 of the logic that would be then implemented in 22 a particular language that you're using but it is not 23 language specific. It's just -- it's supposed to 24 represent the logic. 25 Q. So and this is intended for human reading rather than 80 1 machine reading? 2 A. Exactly. 3 Q. Now, this code enables the sale price of any of the 4 three items to be changed; is that right? 5 A. Absolutely. We could always go in and change the pound 6 amount that's associated with each one of these items. 7 Q. But that would require a change to the source code? 8 A. Yes. That is not ideal. 9 Q. Can we compare this to a data-driven logic approach and 10 look at the code that is written in pseudo-code under 11 paragraph 4.5.15? 12 So, again, the part in bold and italics, under 13 4.5.15, if we could just blow that up. 14 A. I do want to reference table 4.3, which is behind it. 15 The first part of this relies on the reference data 16 that's in table 4.3. 17 Q. Okay, yes, so that's it. We can see both of those, 18 I hope, at the same time. 19 A. Yes. 20 Q. So if you can talk us through this code by reference to 21 the table at 4.3. 22 A. Sure. So, as before, setting the total basket amount to 23 zero, and then I am iterating through the different 24 items that are purchased. So for every product 25 purchased, I'm going to first look in that table, in the 81 1 table 4.3, to see if I find that particular item. So, 2 for instance, if I'm looking for a hammer, I see that 3 there's a hammer in that table and I can see that the 4 price for that hammer is £5. So if I find that product 5 I'm going to multiply the quantity by the price, and add 6 it to the basket. 7 In a similar fashion, when I get to the 8 screwdrivers, I'm going to take the quantity of 9 screwdrivers and multiply it by the £7 that's associated 10 with the screwdrivers and then do the same thing for the 11 pair of pliers. I'm going to multiply that by the £6 12 for the pliers by the quantity that was purchased. Each 13 time I do that, I'm adding it to the total basket 14 amount, and, at the end of it, I should have come up 15 with the same total that the prior version came up with. 16 Q. But the difference here is that any price changes can be 17 made by an alteration to the product master table with 18 no need to fiddle with the code? 19 A. That's exactly right. 20 Q. Is there anything else you want to say or you say in 21 your report here that, since price changes can be 22 frequent, it is appropriate to use the latter method 23 rather than the former? 24 A. Yes. So data-driven logic or data-assisted programming 25 allows for adroitness in maintaining the code, or 82 1 maintaining the system, because you are not required -- 2 whenever you make a change to source code, and try to 3 deploy that, you should be going through the testing 4 process to do that. To the extent that you can remove 5 items like price changes from the code to more of a data 6 driven technology, that reduces the amount of items that 7 need to be tested. Because you already know that the 8 code works. You simply need to make sure that you are 9 maintaining that table correctly without requiring going 10 through the whole testing cycle for any new code. 11 So, in the first example, if we changed prices, 12 I would have to go change the code and then, in theory, 13 I should have to test that code again to make sure the 14 works right. In the second example, I simply need to 15 make sure that someone is in charge of maintaining this 16 table correctly. I'm never changing the code, therefore 17 I don't have to go through the testing process just 18 because there's a price change. 19 Q. In the first example you said that you would, in theory, 20 have to go and retest. Is that because the alteration 21 to the code may have unintended consequences for other 22 parts of the operation of the system? 23 A. Yes. Now, in this simple example -- this is such 24 a simple example that I couldn't imagine what this 25 change could do but there are many more complex issues 83 1 that can be handled by data-driven logic and those 2 certainly have the opportunity to introduce more issues, 3 or more opportunities for error, and would require 4 a full battery of testing every time the code changed. 5 Q. You've explained to us the high-level design of the 6 Horizon System. Can we turn to the high-level structure 7 of the Horizon System. 8 You explain that there are number of ways in which 9 you might approach the description of a system like this 10 but you have, for simplicity, characterised the system 11 into four main components. Do we see those listed in 12 4.5.17? 13 A. Yes. 14 Q. Can we go over the page, please, to figure 4.4, and just 15 blow up that figure so it takes up the page. Using that 16 figure, which sets out the component elements of Legacy 17 Horizon, deal with the four components in summary form 18 first, and then go into depth on three of the four 19 components. 20 So starting from the bottom of the table, please, 21 component A. Is component A or does component A consist 22 of the counter peripherals, so the parts of the system 23 that were located in the branch, consisting of both 24 hardware and software? 25 A. Yes. 84 1 Q. Then moving up to component B on the left-hand side of 2 the table, is that the communications network? 3 A. Yes. 4 Q. This is, in summary, functionally the same as what we're 5 used to nowadays, an Internet connection but, in fact, 6 back then was either the dedicated ISDN line that we 7 spoke about or sometimes a satellite link? 8 A. That is correct. 9 Q. Can you just explain what an ISDN line was? 10 A. It effectively was -- it was a communication mechanism, 11 a piece of hardware that was offered by telecom 12 companies that allowed a connection to be made to push 13 data through from satellite offices to a central office. 14 It was a communication mechanism. 15 Q. By satellite offices, you don't mean offices with 16 a dish -- 17 A. Sorry -- 18 Q. -- you mean remote branches? 19 A. Yes, branches. 20 Q. Thank you. This element, component B, is not something 21 we are going to explore further, other than to say that 22 you understand that the communications network was 23 provided by a combination of services given by BT and 24 a company called Energis; is that right? 25 A. That's correct. 85 1 Q. Moving to component C on the right-hand side of the 2 table here, the messaging system, did this comprise, 3 again in summary form for the moment, the software and 4 protocols responsible for encapsulating data and for 5 permitting communication between branches and the 6 Horizon campuses, as you call them? 7 A. Yes. 8 Q. Then lastly D, the campuses, the Horizon campuses. Can 9 you explain in general terms, in summary level at the 10 moment, what the campuses are and why they're called 11 campuses? 12 A. They're called campuses because there were two of them, 13 one at Bootle and one at Wigan. They were data centres 14 operated by ICL and they acted as the collector and 15 manager of all of the transactions that were generated 16 at the branches. 17 Q. Before looking at components A, C and D in a little more 18 detail, you make the point in paragraph 4.5.19 that the 19 system was designed to operate with an available network 20 connection, i.e. in an online mode, but was also 21 designed to operate without such a connection, 22 an offline mode. This is something you mentioned 23 15 minutes or so ago. Were there exceptions to that 24 which prohibited the system from operating other than in 25 online mode? 86 1 A. The -- most services were able -- or most transactions 2 were able to be conducted in both the online and offline 3 mode with the exception of two: the national banking 4 services -- 5 Q. The network banking services? 6 A. Sorry, network banking services and the debit card 7 services. The reason these two services were not 8 allowed to operate in the offline mode is there had to 9 be a handshake confirmation that the transactions 10 related -- that a confirmation that these transactions 11 were allowed by the actual clients before they could be 12 transacted. So, in other words, if I was trying to 13 withdraw money from a bank, the bank needed to tell the 14 Horizon counter that they gave permission to withdraw 15 that money, to the extent that the counter wasn't in 16 offline mode, they could not communicate with the bank, 17 therefore that service was not available. 18 Q. Thank you. Can we turn, then, to the first element of 19 Legacy Horizon, component A. Can you please run through 20 the main physical components of the IT system within 21 component A located within the branch. This is 4.5.21 22 of your report, please? 23 A. Certainly. First, there was the counter. This was the 24 PC that had the touchscreen on it, that the SPMs and 25 their employees would have operated the Horizon System 87 1 through. So that encompasses both A and B subparagraph 2 on this section. They had a keyboard, a barcode reader, 3 weigh scales for weighing postal items, a tally roll 4 printer, PIN pads and an A4 printer. 5 Q. Just looking at those in some more detail, the keyboard: 6 you describe this as a specialised financial keyboard 7 with a magnetic strip reader and smartcard reader on it. 8 So this was a bespoke design? 9 A. Yes, that is correct. 10 Q. The tally roll printer, what is meant by "tally roll"? 11 A. It was the printer used for printing out customer 12 receipts as well as some of the reports that were 13 designed by the Horizon System. 14 Q. You say that some branches had but a single counter, and 15 by a "counter", do you mean the elements that you've 16 just described? 17 A. Yes. 18 Q. About 46 per cent of all branches had a single counter 19 and I think you tell us that the figures you have seen 20 suggest that 33 per cent of branches had two counters 21 and the remainder three or more counters; is that right? 22 A. That is correct. 23 Q. Is it right that, in order to use a counter, an SPM, 24 a subpostmaster, would need to log in to the counter 25 using their assigned username and password? 88 1 A. Yes. 2 Q. Can we look, please, at the figure on page 35, 4.5, and 3 just blow up the figure at the top of the page there. 4 Thank you. Is this right: your depiction of a set-up in 5 branch where there existed a single counter? 6 A. Yes. 7 Q. Can you just talk us through it, please? 8 A. Certainly. So you'll see the components that we already 9 described. You have the counter, the PIN pad, the weigh 10 scales, the monitor, the keyboard, the tally roll and 11 the barcode reader and the A4 printer. All of those are 12 connected to the PC or what we're going to refer to as 13 the counter, specifically. The counter -- you'll also 14 notice that the counter, it's called the Gateway PC and 15 this will make more sense when we get to the 16 multi-counter description. 17 But every branch had a Gateway PC, to the extent 18 that the branch had a single counter, that single 19 counter was the Gateway PC. The purpose of the 20 Gateway PC designation is that is what communicated with 21 the campuses and you'll notice that you see that the 22 two-direction arrow that connects the Gateway PC and the 23 LHITS campus. Some of the components were used by the 24 customer, some components were used by the SPMs and the 25 weigh scales could be used by either. 89 1 Q. Can we turn to the position on multi-counter branches, 2 please. Can we look over the page, please, at table or 3 figure 4.7, and blow that up, please. Is this your 4 depiction of the position in multi-counter branches? 5 A. It is, yes. 6 Q. Again, can you talk us through it, please? 7 A. Certainly. So the big distinction between the 8 single-counter branch and the multi-counter branch is 9 the fact that there are multiple counters at the 10 multi-counter branch but the difficulty or the 11 complexity that this presents for this particular branch 12 is that there there's still only one of the counters 13 acting as the gateway between the LHITS campuses and the 14 branch. 15 All of the other counters that are not considered 16 the Gateway PC have to be connected to the Gateway PC 17 and you can see that there is an extra box in this 18 diagram that is labelled "Hub connecting the Counters", 19 and that is the extra piece of technology that needed to 20 be introduced to connect -- make sure that all the 21 counters could communicate with each other and, 22 importantly, communicate with the Gateway PC because the 23 Gateway PC counter was the communication hub to the 24 LHITS campus, which is important because that's what 25 transmitted all the transactions. 90 1 Q. There was a function or a feature of this system, 2 I think, that allowed a subpostmaster to transfer 3 an open session when dealing with transactions between 4 counters; is that right? 5 A. That is correct. 6 Q. Can you just explain what that feature was? 7 A. So, the -- this goes back to anticipating, you know, 8 problems or operational situations where perhaps an SPM 9 or one of their clerks had started a transaction with 10 a customer at a particular counter and, for whatever 11 reason, needed to switch to a different counter. The 12 functionality that you just described is part of the 13 design of the Horizon System, in that you could move 14 that particular session from counter A to counter B, if 15 you found that necessary. 16 Q. Thank you. Can we turn to the software, please. That 17 table can be taken down. I think it's right that 18 Horizon used the Windows NT operating system, "NT" 19 meaning "new technology" -- is that right -- 20 A. That is correct. 21 Q. -- which, of course, it was at the time. You explain 22 that users were prevented from directly accessing 23 Windows; is that right? 24 A. That is correct. 25 Q. What's the importance of that preventative step? 91 1 A. The -- all of the functionality that the SPMs at the 2 branch -- SPMs and clerks at the branches, all of the 3 functionality that they needed was through the Horizon 4 software. There was no need for them to ever get to the 5 operating system level, which is what Windows NT was. 6 The set-up configured each one of the counters so that 7 as it booted, the Horizon System came up or the repost 8 that -- the screens, the user interface, the 9 touchscreens, would come up, and there would be no 10 ability for the user to get to Windows. 11 Why is that important? If all of the functionality 12 that the SPMs needed to operate the Horizon System 13 existed within the Horizon application, there is no need 14 to go anywhere else because only bad things could happen 15 at that point. If you had access to the operating 16 system, you could change configurations, you could do 17 a lot of things that would deteriorate the PC and 18 perhaps make the Horizon System not work correctly. So 19 I suspect that is why no access to the operating system 20 was part of the design. 21 Q. So the system was configured to prevent subpostmasters 22 having access to a dot prompt? 23 A. Yes, or -- yes. 24 Q. Is that what you (unclear: overspeaking) a little bit? 25 A. You could say the dot prompt, yes. It would restrict 92 1 their access to the dot prompt. 2 Q. Thank you. The Windows NT operating system, how old was 3 that at the time, in 2000, can you recall? 4 A. It was aging by the time the 2000s came around. 5 I believe it was introduced in the mid-'90s, so it was 6 mature. 7 Q. Do such operating systems have planned obsolescence 8 within them? 9 A. Yes, they do. 10 Q. Do you happen to know what the planned data obsolescence 11 was for Windows NT? 12 A. I believe that that was the mid-2000s that the planned 13 obsolescence for Windows NT was. 14 Q. You tell us that rather than being allowed directly to 15 access Windows, subpostmasters, when they logged on, 16 were sent automatically to a piece of software that had 17 been specifically configured for the Post Office, The 18 Riposte Desktop; is that right? 19 A. Yes. 20 Q. Can you explain to us what The Riposte Desktop was? 21 A. The Riposte Desktop was the user interface that the 22 clerks at the branches would have interacted with to 23 operate the Horizon System. 24 Q. You say this was largely based, ie that system, the 25 Riposte Desktop, on a commercial product named Riposte 93 1 from the Escher Group. What do you know about the 2 Escher Group? Who were they? 3 A. I know they were a software development group that 4 specialised in retail software. 5 Q. They were a separate company from ICL Pathway Limited 6 and Fujitsu Services Limited; is that right? 7 A. That is correct. 8 Q. You go on to say that the counter user interface, or UI, 9 user interface, was designed to be as simple and 10 intuitive as possible, and was specifically tailored for 11 use in a retail environment. So the counter user 12 interface, that's the same thing as The Riposte Desktop? 13 A. Yes. 14 Q. You say that the intention was that the subpostmaster or 15 clerk had no requirement to type; is that right? 16 A. That is correct. 17 Q. So what was done instead of typing? 18 A. There's a touchscreen. They could use the touchscreen 19 as well as the card readers and the PIN pads to enter 20 information into the Horizon System. 21 Q. Some transactions, is this right, were initiated not by 22 touching the screen but by an activity on a peripheral? 23 A. That's correct. 24 Q. Such as swiping a magnetic card or reading a barcode 25 using the barcode reader? 94 1 A. Yes. 2 Q. Can we display over the page, please, figure 4.8. Can 3 we blow up that at the top of the page, please. Is this 4 a screenshot of the user interface on Legacy Horizon? 5 A. Yes. 6 Q. Can we see that it's split into two parts about 7 two-thirds or four-fifths away across the page, from 8 left to right. We can see a line going up and down the 9 screen. Is that the division between the left-hand and 10 the right-hand part? 11 A. Yes. 12 Q. There are a series of menu buttons on the left-hand 13 side. Were those menu buttons or tiles available to 14 press in the context of a particular transaction? 15 A. Yes. 16 Q. Is it right that some of them sometimes displayed a stop 17 sign, preventing them from being depressed and actioned 18 because they weren't available for that particular 19 transaction? 20 A. Yes. 21 Q. Then, on the right-hand side, is what you describe as 22 a "stack", showing the purchases made by the customer; 23 is that right? 24 A. That is correct. 25 Q. Is there anything else you wanted to say about the user 95 1 interface, the screen on Legacy Horizon? 2 A. Um, you know, the intent and, you know, whether this 3 intent was fulfilled, is to the taste of the user. The 4 intent was to make it as simple as possible, make it 5 a graphical interface and to reduce the chances for 6 error. But beyond that, no, I don't have anything else 7 I want to say about this screen. 8 Q. Thank you. You speak in your report of the concept of 9 a stock unit, and you describe it as being an important 10 concept. Can you explain what a stock unit is and why 11 it is important? 12 A. Certainly. So for the managerial accounting at each one 13 of the branches, a stock unit was a concept or an 14 abstraction of how to -- it gave the branches the 15 ability to organise themselves however they felt fit. 16 The stock unit -- there could have been one -- so we'll 17 get to this when we get to the rollover but, 18 essentially, a stock unit was -- could represent 19 a particular till. It could represent a particular till 20 for a particular amount of time. But essentially, it 21 was a way to account for your stock and your cash at 22 that particular branch for a particular period of time 23 and, like I said, it allowed the SPMs to divide up their 24 stock and cash amongst their different clerks, as well 25 as to limit the amount of time, you know, cordon down 96 1 the time that that stock and cash was in the possession 2 of someone. 3 It would appear I'm not being clear here. It's 4 an abstract concept that basically allows you to divide 5 up the cash and stock within a particular branch as the 6 SPM felt fit, either across people or across time 7 dimension. 8 Q. Thank you. You say in your report that it's a way of 9 managing cash and stock. They can be allocated to 10 an individual subpostmaster on a medium-term basis or to 11 an individual counter clerk, and then that person is 12 responsible for ensuring that the stock unit balances at 13 the end of the period, whether that be a week or however 14 long the stock unit is allocated to them for. 15 A. Yes. 16 Q. You say stock units are assigned identifiers, such as 17 "DD" or "AA". Can you just explain what you mean there? 18 A. Certainly. So if there are multiple stock units that 19 are being issued by the SPM, they need a unique 20 identifier. "DD" and "AA" are just examples of how you 21 might uniquely identify a stock unit. 22 Q. You say that subpostmasters can transfer stock between 23 stock units using a function on the counter. Stock 24 units can be individual or can be shared between 25 multiple counter clerks. In some circumstances, the 97 1 subpostmaster may choose to allocate a stock unit to 2 a certain specified stock, such as Lottery Scratchcards; 3 is that right? 4 A. Yes. 5 Q. To what extent was this left to the subpostmaster to 6 determine, ie the object of the stock unit, lottery 7 Scratchcards; the duration of the stock unit, a week, 8 two weeks; or the number of people involved, a specific 9 counter clerk or a specific counter? 10 A. This, the concept of the stock unit, theoretically 11 provided a lot of flexibility for the SPM to use at 12 their discretion. 13 MR BEER: Thank you. 14 Sir, it's a couple of minutes before 1.00. I was 15 about to move to the topic of modes. Can we pick that 16 up at 2.00, please? 17 SIR WYN WILLIAMS: Yes, of course. We will see everyone at 18 2.00. Thank you very much. 19 MR BEER: Thank you, sir. 20 (12.58 pm) 21 (The Luncheon Adjournment) 22 (2.00 pm) 23 MR BEER: Good afternoon, sir. Can you see and hear me? 24 SIR WYN WILLIAMS: Yes, I can. Thank you. 25 MR BEER: Thank you very much. 98 1 Mr Cipione, we were just turning to paragraph 4.5.30 2 at the foot of page 37 of your report, where you speak 3 about modes, ie the modes in which the counter could 4 operate. Could you describe the various modes that you 5 speak to there, please? 6 A. Certainly. So the concept of a mode allows the 7 different functionality of the Horizon System to be 8 accessed by the user, the one that everyone will be most 9 familiar with is serve the customer, conducting the 10 business of selling stamps, performing the transactions 11 of the Post Office stock and cash. 12 Q. Just stopping there, if we go back to the screenshot of 13 the user interface at the top of the page, please. If 14 we just see there as this user interface is displayed in 15 this screenshot, it is, in fact, displayed in the mode 16 of "Serve Customer"? 17 A. That's correct. 18 Q. We can see that on the tile at the top left-hand -- 19 I keep calling them "tiles". Is "tiles" the right 20 expression or not? 21 A. Tile, button. I'm not sure exactly how they referenced 22 it internally. 23 Q. In any event, this one is displayed in the "Serve 24 Customer" mode; is that right? 25 A. Yes. 99 1 Q. You were telling us that that was the mode that people 2 might be most familiar with, SC, "serve customer". Can 3 you tell us about the other modes please? 4 A. So other modes include all of the back office processing 5 that might need to be done, such as adjusting stock, 6 REMing in or REMing out supplies and doing various 7 housekeeping. So, essentially, there is the main mode 8 or what I would describe as the main mode, which is the 9 "Serve Customer" and a lot of other supporting modes to 10 make sure that all of the housekeeping items or all of 11 the ancillary work that needs to be done to manage the 12 particular branch is available. 13 Q. When one is in a mode, the other tiles that are either 14 displayed or available within that mode may increase or 15 decrease; is that right? 16 A. That is correct. 17 Q. So the extent to which you can use the other buttons is 18 affected by the mode that you have selected. 19 A. Yes. 20 Q. You mentioned in your answer there the REM-out supply 21 division, ROSD, and REM-in supply division, RISD; can 22 you explain the concept of REMing in and REMing out, 23 please? 24 A. Yes. So the stock and cash at a particular branch needs 25 to be -- could be undersupplied or oversupplied, 100 1 depending on the activity that's occurred at that 2 branch. If it's undersupplied, the branch has the 3 availability to reach out to POCL to provide more cash 4 and/or stock as they receive that. That would be 5 considered REMing in cash or stock. Alternatively, they 6 might have collected too much, as far as cash or stock 7 is concerned, at which point they want to return that to 8 POCL or the Post Office, and that is REMing out. 9 Q. You describe in your report that there are three types 10 of transactional data in the system: manual entries, 11 transaction corrections and Fujitsu entries. Can you 12 explain each of them, please? 13 A. Certainly. Manual entries would be the transactions 14 that occur as the user is operating the Horizon counter. 15 So as they're conducting the business of stamps and 16 other services available through Horizon, that is in 17 the -- that's what is considered manual transactions and 18 that would be most of the activity that one would expect 19 to understand about the activity at that particular 20 branch. 21 There are instances where some extra transactions 22 need to be entered into the system, either to correct 23 errors or to adjust for different situations and those 24 two -- one is called a transaction correction or a TC, 25 and that is generated by the Post Office; and the second 101 1 one would be when Fujitsu entered transactions to do the 2 same type of corrective error. So one was sourced from 3 the Post Office and one was sourced from Fujitsu. 4 Q. You mentioned lastly, in relation to this component, 5 component A, of Horizon, that Legacy Horizon was coded 6 in Visual Basic, C and C++. 7 A. Yes. 8 Q. It also utilised Oracle development tools and, as you 9 have said already, the Riposte product. Where in your 10 categorisation of the types of software do those species 11 of software sit? 12 A. I'm sorry, could you repeat the question? 13 Q. Yes. You earlier categorised software into different 14 types. 15 A. Oh, okay. 16 Q. Where in that categorisation do those three sit? 17 A. Yes, so Visual Basic, C and C++ are all programming 18 languages. So I would include those in the application 19 development category. The Oracle development tools, 20 Oracle -- these development tools would have been 21 related to the relational database management system, 22 the database that Oracle provides. So I would put that 23 in DBMS. The Riposte product, that is an application -- 24 well -- that's a little bit of a mixed bag. 25 Q. A hybrid? 102 1 A. Yes. So the Riposte product is an application but it's 2 an application that was conformed to the Horizon 3 product. So you could see there it was an application 4 or an application development tool. 5 Q. Thank you. Can we turn to component C of Legacy 6 Horizon, please, the messaging system. You say in the 7 first line of paragraph 4.5.34 that the counter uses 8 a messaging infrastructure provided by a system called 9 Riposte. Can you explain what you mean, please, by 10 a "messaging infrastructure"? 11 A. Yes. So a messaging infrastructure is a way of 12 encapsulating data in a structured manner so it can be 13 communicated in bits and bytes to other parts of the 14 system. So it's data -- essentially it's data with 15 a lot of structure around it so that other components of 16 the system can understand what that data means. 17 Q. You say that this was called Riposte, developed by 18 Escher. You say: 19 "Everything that Riposte handles is stored as 20 a message. Messages are constructed using a format 21 known as Attribute Grammar." 22 Then this sentence: 23 "This is a self-defining and nested record format 24 that is technology independent." 25 When I first read that, I started thinking about 103 1 next summer holiday. Can you explain what you meant, 2 please? 3 A. It's a method. So the Attribute Grammar is a method of 4 applying a structure to certain data -- to data. So 5 maybe an example might be -- make it easier. 6 Q. Thank you. 7 A. So let's say I wanted to send an email to someone, okay? 8 Part of the components of that email would be who 9 I wanted to send it to, part of the component would be 10 the date I'm sending it, part of it might be the subject 11 line and part of it might be the body of the email. 12 Using this messaging type structure, as I wrote that 13 email and as it came off of my email system, it would 14 go -- as it was being transmitted there would be a tag 15 that says, "This next bit of information represents who 16 I'm sending it to and stop there". Then there would be 17 another tag that says, "This is the subject", and then 18 there would be another tag that says, "This is the 19 date", and then there would be another tag that says, 20 "This defines the body of my email". 21 All of that would be wrap inside perhaps some other 22 tags that would help coalesce all that information, 23 identify it as all being identified to each another and, 24 as a package, that could be sent and then interpreted by 25 another component of the system that would know -- that 104 1 would understand that structure and would be able to 2 consume that information. 3 Q. You say they are a "tree-like structure" and could be 4 considered a "proto-markup language". 5 A. Yes. 6 Q. Can you explain what you mean by that, please? 7 A. So the example I just gave was anticipating that 8 question, so a markup language is simply, number 1, 9 defining that this is the beginning of my message and 10 then it could have many elements in it. It could have 11 one element in it. There would be indicators within 12 that markup saying, this is -- "You should expect five 13 elements or you should expect two elements or you should 14 expect elements until you see an end-of-element marker". 15 But within each one of those markers would be the 16 specific information that you're trying to send and it's 17 identified very systematically and very structured. 18 It's easy to understand by a program but not necessarily 19 easy to read by a human being. It's an efficient way of 20 categorising and organising information and delivering 21 it for further consumption by other components. 22 Q. Can we see this in action with the hammer, screwdriver 23 and pliers -- 24 A. Yes. 25 Q. -- and go on to 4.5.36 over the page. So, as you remind 105 1 us, the purchases of two hammers, three screwdrivers, 2 and one pair of pliers, using Attribute Grammar, you say 3 the tree-like structure might use the following 4 relationships, yes? 5 A. Exactly. Exactly, so -- 6 Q. Can you talk us through 4.5.36, please? 7 A. Yes. So you'll see that at the beginning of this 8 structure I'm indicating that this is a sales 9 transaction. So it's marked there. Everyone that 10 receives this will know that this is a sales 11 transaction. 12 Q. Again, this is written in pseudo-code -- 13 A. This is a pseudo-code. 14 Q. -- for humans to read rather than the machine. 15 A. Yes, exactly, exactly. So underneath -- so the first 16 thing that this particular tree-like structure wants to 17 communicate, in addition to that it's a sale 18 transaction, is who is the customer. So that's the 19 second element you see there. Then you'll see that 20 there is an indicator of how many items were purchased 21 by this customer, just to give an indication. So there 22 can be internal integrity checks to be done on what was 23 built into this particular message. 24 Once we get past the count of the items purchased, 25 we get to item 1, and this is more of the tree-like 106 1 structure. Item 1 will have a quantity and a price and 2 a total purchase amount related to item 1. Then we move 3 on to item 2, and it will also have a quantity, a price, 4 and a total purchase amount. That could go on 5 infinitely or it could go on once, it really just 6 depends on what the actual transaction is but the 7 flexibility of this particular type of structure doesn't 8 require you to have just one or just five or just ten. 9 It allows -- it's a flexible way of storing 10 information and, at the end of this all, you see that 11 there's the total basket purchase amount. 12 Q. Looking at subparagraph 37 there, you translate that 13 into our example of two hammers, three screwdrivers and 14 one pair of pliers. 15 A. Yes, exactly. 16 Q. Can you just talk us through that? It may be obvious, 17 given what you've just said. 18 A. So we see that the sales transaction, there's a start 19 and an end marker on the sales transaction. You can see 20 that that is the giant wrapper around this particular 21 message. The first element within that particular 22 wrapper is the customer name, which I chose "John Doe" 23 as the customer and, past that, you'll see that this 24 message contains three items that were purchased. 25 The first purchased item name was a hammer with 107 1 a quantity of two and a price of £5 which totals to £10 2 for that particular item purchase. The second item is 3 a screwdriver, quantity of three, with a price of £7, 4 which results in a total purchase amount of £21. The 5 third item is pliers with a quantity of one, a purchase 6 price of £6 and a total purchase amount for item 3 of 7 £6. Then it sums up all of the purchase amounts as £37 8 and ends. 9 Q. Thank you. Can we go forward to page 166 of your 10 report, please, which is Appendix B. 166, thank you. 11 Can we see here an example of Attribute Grammar actually 12 written in code? 13 A. Yes. 14 Q. This isn't, I think, the hammer, pliers or screwdriver 15 example? 16 A. It is not. Like I indicated earlier, this is not human 17 friendly to read. You would have to understand the 18 definition of the structure, see what elements were 19 available within the structure to properly interpret 20 this. But anyone using this particular Attribute -- if 21 this Attribute Grammar was the standard for this 22 particular system, every component of the system should 23 be able to interpret this properly. 24 Q. Would the coder be able to interpret it? 25 A. It depends on what coder we're talking about. Not all 108 1 current coders would need to be able to interpret this 2 because there could be a group of coders that are 3 designated simply to do the translation of this 4 information. So there might be one group of like four 5 or five coders whose complete responsibility is to make 6 sure that this structure is understandable to all 7 systems and to write the code that can both encode and 8 decode information passed in this format. 9 Q. Lucky them! 10 A. Yes. 11 Q. Can we go back, please, to page 39 of your report and 12 paragraph 5.39 at the foot of the page. You say: 13 "Normal transactions at the Counter take place 14 within a customer session. Each physical transaction 15 with the customer ... results in the creation of one or 16 more messages depending on the complexity of the 17 transaction." 18 Then you say this: 19 "For example, a stamp sale has one message, and 20 a postal order results in two messages (one for the 21 postal order and one for the fee). None of these 22 messages is normally written to the message store until 23 the customer 'settles' the session." 24 Can you just explain this, please? 25 A. Yes, so what this is indicating is that, as the clerk is 109 1 transacting the business on the counter and adding in 2 different items to the transaction, it's not recorded 3 until -- there's no need to record this information 4 until the transaction has become official by settling 5 it, by receiving some form of payment, so that's what 6 I'm trying to communicate here. 7 Q. So the stamp results in one message but the postal order 8 results in two messages. Can you just explain that? 9 A. Certainly. So for some transactions there might just be 10 one item that needs to be recorded in a message and for 11 other -- so the stamp. That's a fairly simple, 12 straightforward action. I want to pay £5 for a set of 13 stamps. In the Horizon System, that's just one 14 transaction. It's a simple -- I've sold this book of 15 stamps for £5. 16 However, for accounting purposes or managerial 17 purposes, other types of transactions might need to 18 segregate, you know, for instance, the postal order. 19 A postal order might have two components where one is 20 the amount of the postal order and then one is the 21 amount for the fee that is being charged to issue the 22 postal order. So, in that instance, the Horizon System 23 is recording two messages. 24 Q. You say that a key feature of each session is that they 25 are zero sum, that's to say that the debits and credits 110 1 of the transactions must sum to zero; is that right? 2 A. Yes. 3 Q. Over the page, please, to 4.5.40. On completion of 4 a session by a clerk then the resulting transactions are 5 saved locally on the counter, you say, in something 6 called the "counter message store", also known as the 7 "Riposte message store"; is that right? 8 A. Yes. 9 Q. After that sentence, you say: 10 "Where there is more than one Counter in an Outlet, 11 the Riposte Message Store is replicated across all the 12 Counters. Where there is only one Counter, the Counter 13 contains two mirrored disks, one fixed and one 14 exchangeable, so that the message store can be recreated 15 on a replacement Counter if necessary ..." 16 Can you just explain that, please? 17 A. Yes, so this harkens back to the issue we were talking 18 about very early in the discussion this morning about 19 the difficulties of systems in the time period we're 20 talking about. One of them is making sure that there's 21 the proper amount of redundancy of data to the extent 22 that there's a hardware failure. The way the Horizon 23 System handled the possibility of hardware failure -- 24 when I say "hardware failure", I really mean that maybe 25 a hard drive goes bad, because all of this information 111 1 that is being stored locally is being stored on a hard 2 drive. 3 In a single-counter branch -- well, let's talk 4 about -- in a multi-counter branch all of the counters 5 can communicate with each other through their hub or 6 their router and that allows the redundancy to be across 7 different counters. So as a message store on one 8 counter is created, Horizon creates a redundancy of that 9 message store on the other counters, in case of hardware 10 failure on the original counter. 11 For a single-counter branch that multi-counter 12 redundancy is not available to them. So for 13 a single-counter branch, the way that they account 14 for -- the way they make sure that there is a redundancy 15 availability is to have two hard drives within the 16 counter mirroring each other, so that way if one hard 17 drive fails, they always can revert to the second hard 18 drive to recover the information. 19 Q. You explain in subparagraph 41 the data replication 20 facility that Riposte had. Can you explain that to us, 21 please? 22 A. So this is more aligned with the fact that the 23 communication system might not be online at the time 24 that the transactions or the messages are being stored. 25 So remember that the transactions, you can think of that 112 1 as a real world term, but the Horizon digital equivalent 2 to that is a message. So to the extent that there is no 3 communication between the counter and the campuses, the 4 Horizon System, at least, you know, in the late '90s 5 allowed for an accumulation of all of these messages, if 6 they knew that they were in an offline position, to 7 continue to accumulate those until the connection was 8 made again. 9 Then the replication facility would work through all 10 of those stored messages, send them back to the campus. 11 Once it got confirmation from the campus that all of 12 those were restored, it finished the process. But that 13 whole check and balance was just to make sure that the 14 campuses received all of the messages that were stored 15 at the counter while it was in an offline position once 16 it came back online. 17 Q. Thank you. In that subparagraph 41, you say that the 18 counter's messages would be synchronised with a version 19 of the message store saved on the Legacy Horizon 20 campuses in their correspondence servers. Can you just 21 explain what the correspondent servers were? 22 A. Certainly. The correspondence servers were the 23 communication facility at the campuses that basically 24 allowed for the data traffic to move between the 25 counters and the rest of the operations at the data 113 1 facility or at the campus. 2 Q. Okay, we'll come back to those in a bit more detail when 3 we get to component D. 4 A. Yes. 5 Q. Lastly, in subparagraphs 42 and 43, you explain 6 reference data, please. Can you explain to the Chair 7 the three categories of reference data that you mention 8 there? 9 A. Yes. So the reference data -- I broke up the reference 10 data into the three categories. So the reference data 11 could have been derived from POCL or the Post Office. 12 This might be -- like, for example the hammer, 13 screwdriver, plier list, to the extent that the POCL had 14 a list of prices for all of their products, their stamps 15 and what not, that would be an example of reference data 16 coming from POCL. 17 There was also similar information coming from 18 POCL's clients, you know, price lists and other 19 information that the client transactions relied on to 20 process correctly. 21 Q. So that's the DWP or somebody like that? 22 A. Exactly, exactly. 23 Finally, there's reference data that was more behind 24 the scenes reference data. This is reference data that 25 the Horizon System used internally to make sure that the 114 1 Horizon System was operating efficiently. So this would 2 be the -- this would be a more complicated example of 3 a dataset -- a data-driven logic. The Horizon design 4 had a lot of data-driven logic for, for instance, how 5 the menus showed up. What menus were available in the 6 different modes on the user interface on the counter. 7 That would be an example of some information that was 8 actually kept in some reference data and not necessarily 9 coded in the source code. 10 Q. All of those three categories of reference data, they 11 were saved locally on the counter in the counter message 12 store; is that right? 13 A. Yes. 14 Q. That localised copy, is this right, is one of the 15 features that enabled trading to carry on, an operation 16 to carry on, even if connection with the campuses was 17 lost? 18 A. Yes. 19 Q. Can we turn, then, to category or component D, please, 20 the Horizon campuses. Can we look over the page at 21 page 42 to the table at 4.10, please, and just highlight 22 or blow that up, please. Thank you. 23 Could you talk us through the Legacy Horizon 24 campuses, the layers, from the bottom upwards, please, 25 starting with the counter layer -- 115 1 A. Yes. 2 Q. -- which you've already dealt with. 3 A. Yes. So we just talked about the counter layer but, 4 essentially, all of the information is stored in 5 messages. All the transactions are stored in messages 6 at the counter layer, and that's generally what's being 7 communicated to the campuses, and that's being 8 facilitated by what is labelled here the "Correspondence 9 layer". That layer is -- that is the layer responsible 10 for receiving the information into the campuses as well 11 as pushing the information from the campuses back down 12 to the counters. 13 Q. So that's essentially the next layer up. The 14 correspondence layer is the communications of the 15 network; is that right? 16 A. It's the piece of the -- it's the piece of the campus 17 that communicates with the counters. Yes. 18 Q. You say in your report that that layer and the counter 19 layer, shared the use of the Riposte message service, 20 RMS, which was a message storage and replication 21 mechanism running on both the correspondence servers and 22 on the counter; is that right? 23 A. Yes, that's right. 24 Q. You say that supported a shared distributed message 25 store that ensured that information penetrated at the 116 1 counter is replicated at the campuses and vice versa -- 2 A. Yes. 3 Q. -- and that those Riposte mechanisms interact directly 4 with the agent layer, a specialised Riposte Archiver 5 running on the correspondence servers was used to ensure 6 that all Riposte messages are written to tape for audit 7 purposes. Can you just explain that, please? 8 A. Yes. So the agent layer, I would say that that would be 9 your universal translator for the use of all the 10 information, both operationally and financially, once it 11 got through the correspondence layers. So one of the 12 items like you said is -- you know, it made the 13 information available to be written to archive or to 14 tape. It also translated it into the multiple different 15 consumers of this information that we'll get to in 16 a second, but it was where the translation from the 17 Riposte message store format to whatever format was 18 needed by any one of these connected services appeared, 19 and it will also be able to reverse that translation if 20 one of these other services needed to push information 21 back down to the counter. The agent layer would be the 22 layer that translation occurred at. 23 Q. You say it provides facilities to pass messages directly 24 to third-party clients and to return the client response 25 back to the counter -- is that right -- 117 1 A. Yes. 2 Q. -- and that this layer, the host layer, applies any 3 business rules to the information being received from or 4 sent to the external client? 5 A. Yes. 6 Q. Can you deal with next on the table the RDMS, the data 7 reference management system? 8 A. Yes. We spoke about this a few minutes ago and often 9 throughout the discussion today, the Horizon System 10 relied on reference data to operate correctly and, if we 11 remember back to my theory section, it's very important 12 that reference -- you know, when a system does rely on 13 data-driven logic, it's important that there be a system 14 that ensures the integrity of that data within the 15 data-driven logic. That's -- the responsibility of the 16 RDMS system is to do the maintenance updating and 17 control the integrity of the information that's being 18 used as reference data within the Horizon System. 19 Q. So, as you say, its integrity -- the integrity of the 20 RDMS, is important to the proper operation of Legacy 21 Horizon? 22 A. Yes. 23 Q. Can you next deal with the data warehouse, please. 24 A. Certainly. The data warehouse is a place where -- it is 25 a facility where information is collected for further 118 1 analysis by POCL. There are a number of different 2 analyses or reporting requirements that POCL had. This 3 data warehouse is basically the data management layer 4 that responds to providing information for reporting and 5 analytics. 6 Q. You tell us in your report about the transaction 7 processing system or TPS, which is the next box to the 8 left, at this layer. Can you tell us what function that 9 had, the TPS? 10 A. Yes. So all of the transactions that are collected by 11 Horizon ultimately terminate at POCL -- in POCL's 12 financial systems. The TPS system is what translates 13 the transactions that are collected from Horizon and 14 delivers them to the POCL's TIP system, which is their 15 transaction information processing system, which is 16 what -- it's POCL's internal system for their accounting 17 processing. 18 So, basically, it delivers all of the managerial 19 information, all the managerial accounting system to 20 POCL so they can use it for their financial reporting as 21 well as their managerial aspects of their business. 22 Q. You say that that system, TIP, the final POCL system, 23 was one used by POCL to collect transaction records 24 about all transactions that occurred at branches -- 25 A. Yes. 119 1 Q. -- that such transactions were gathered, in the first 2 instance, by the transaction processing system once the 3 counter had set the end of day marker. Can you tell us 4 what the end of day marker is? 5 A. That's an indicator that says where -- "This is a good 6 set of transactions for this day and we're ready to 7 deliver them to be processed further by POCL" or 8 whatever else needs to process these transactions. 9 Q. You say in your report that Legacy Horizon was not 10 an end-to-end accounting system but the data that it 11 passes to Post Office Counters Limited and its clients 12 had to be sufficient to enable them to balance their own 13 books and to settle accounts between them? 14 A. Yes. 15 Q. I think I have not included, so far, the management 16 information services MIS. Can you explain what that is, 17 please? 18 A. Oh, yes. So this was an adjunct to the data warehouse, 19 to continue to further assist POCL in analysing the 20 data. But this was, I believe, specifically used to 21 detect errors or had been an error detection system for 22 the transactions that were received from the Horizon 23 System. 24 Q. Lastly, it may be obvious, external interfaces in the 25 top right, Post Office Counters Limited and clients. 120 1 Just explain what that's meant to be? 2 A. Yes. So, as you know, there are instances that the 3 Horizon System needed to reach out and speak directly or 4 communicate directly with the external client community. 5 That's what this represents. You know, to the extent 6 that network banking services needed to get 7 a verification of sufficient funds to do a withdrawal, 8 that would be represented here. 9 Q. Thank you very much. I think that's the end of 10 component B and, therefore, the end of the description 11 of Legacy Horizon's three main components? 12 You speak in the next paragraph of your report, 13 4.5.53, which is over the page, about software updates 14 made to Legacy Horizon between rollout in 1999/2000 to 15 the introduction of Horizon Online in 2010. You 16 highlight seven or eight of the most significant of 17 them. Can you just talk us through those? Firstly, the 18 Core Systems Release that was introduced in August 2000. 19 A. Yes, the Core Systems Release introduced the Automated 20 Payment Services, which supported payments by customers 21 to the utility companies, such as British Telecom, 22 electricity and other utility companies, as well as 23 other clients using barcoded bills, magnetic cards or 24 smart cards. It also introduced reconciliations between 25 the Automated Payment Service and the data harvested by 121 1 the APS agents and data harvested by TPS. So a lot of 2 internal reconciliations functionality appears to be 3 added here. 4 I also note that it seems that there was also Core 5 Systems Release Plus released, which I would assume was 6 closely aligned with Core Systems Release. 7 Q. In February 2000, there was an upgrade called 8 Maintenance Release M1 and you say that the main purpose 9 of that was to enhance the CSR+ applications. 10 A. Yes, yes, that's correct. 11 Q. In June 2001 there was a release called S06 Release Day 12 D Rectifications; is that right? 13 A. Yes. 14 Q. That included a fix. We're not in this table or this 15 paragraph looking at all of the fixes that were 16 released; is that right? 17 A. That is correct. 18 Q. In 2002/3, Network Banking Service, NBS, Debit Card 19 Services, DCS, and Data Reconciliation Services were 20 introduced. 21 A. Yes. 22 Q. You describe in your report what reach of those three 23 features are? 24 A. Certainly, sir. The Network Banking Services allowed 25 customers of selected banks and building totals withdraw 122 1 money from or deposit money into their bank accounts. 2 So it allowed a banking option functionality to be 3 accessed through the Horizon System. 4 The Debit Card Service was -- allowed customers to 5 have had the functionality of using their debit cards to 6 pay for goods through the Horizon System. 7 The Data Reconciliation Services related to all the 8 NBS and DCS transactions and helped reconcile them for 9 the different data flows. So it was an internal 10 reconciliation facility. 11 Q. In subparagraph (f) you mentioned a development in 2004 12 when Post Office Limited Finance System was implemented, 13 and you described that as an SAP accounting system. 14 What do you mean by that? 15 A. SAP is -- in fact we used this as an example earlier, 16 SAP provides ERP systems, a part of which are the 17 accounting. It provides an accounting application and 18 it looks like the Post Office bought and installed 19 that -- implemented that in 2004. 20 Q. What do you understand this release to have done? 21 A. This essentially took the TIP communication out of the 22 loop from -- so TPS used to deliver their transaction 23 information to TIP. Now it delivered that transaction 24 information directly to SAP. 25 Q. Thank you very much. You say that in paragraph (g), 123 1 over the page, that at some point in 2004, a program 2 called IMPACT was delivered and this included a series 3 of updates. I think there are six of them that you 4 mentioned; can you just talk us through those, please? 5 A. Yes. So the -- a few of them were changing the 6 processes at the branch. The first one I cite here is 7 there were rollovers, which we'll talk in a bit, but 8 rollovers used to be every week, now they were going to 9 be extended to a four to five-week period. Non-value 10 stock declarations weren't required and weren't required 11 by the stock balancing process any more. 12 A concept of a local suspense account was introduced 13 for the processing of variances. Stock units were no 14 longer allowed to carry discrepancies over and any -- 15 over cash accounting -- over trading periods, and they 16 had -- and any discrepancies that arose out of the stock 17 unit balancing were sent into a local suspense account 18 when the stock unit rolled over. 19 Additional checks were carried out in order for the 20 final stock unit to roll over, which included that the 21 stock unit wasn't allowed to roll over if there were any 22 outstanding transaction corrections, and local suspense 23 accounts had to be settled before the final stock unit 24 could be rolled over to the next traded period. 25 Then, finally, there were changes to the data server 124 1 that were made to reduce the number of times that the 2 message store was scanned to pick up transactions during 3 the balancing process. A Riposte mechanism known as 4 Notifications was added -- was used to add new 5 transactions to existing totals as further transactions 6 were generated, so this was more of an operational, 7 behind-the-scenes update. 8 Q. Thank you, Mr Cipione. Can we turn over the page, 9 please, to page 46 and undertake exactly the same 10 exercise, looking at components (a), (c) and (d), 11 missing out component (b), but in relation to Horizon 12 Online, rather than Legacy Horizon. You say that you 13 understand that the online system, HNG-X, as it was then 14 known, was piloted in late 2009 and rolled out in the 15 spring of 2010. 16 A. Yes. 17 Q. You kindly outlined the reasons for the release of 18 Horizon Online, as you understand them from the 19 documents, to be first reducing the cost of running 20 Horizon. Can you explain that in a little more detail, 21 please? 22 A. Yes. The amount -- I would imagine that the 23 communications cost or the old -- the inefficient 24 communications infrastructure and all of the related 25 hardware redundancy that was required to support that 125 1 old technology was rather costly. With the online 2 system, there wasn't a need for much of that redundancy, 3 and that was one of the factors that helped reduce cost 4 in the Horizon Online system, as opposed to the Horizon 5 Legacy system -- or Legacy Horizon system. 6 Q. Secondly, you understand from the documents, the second 7 reason to make the change was in order to take advantage 8 of improved communication technology reliability; is 9 that right? 10 A. That is correct. 11 Q. You say that that change meant it was feasible to have 12 branches that were online for the vast majority of time. 13 That benefitted a wider change in the business as 14 an increased proportion of transactions involved NBS and 15 DCS. Can you remind us what they were? 16 A. Yes, sir, the banking system and the debit card system. 17 Those were the two bits of functionality that were 18 required -- that required Legacy Horizon to be in online 19 mode to operate correctly. With Horizon Online, it was 20 assumed that the communication facility would be 21 persistent indefinitely and it allowed for that facility 22 to be available all the time -- or those facilities. 23 Q. Thank you. The third driver that you understand to have 24 operated, from the documents, was in order to simplify 25 the design of the user interface. Can you explain what 126 1 you understand by that? 2 A. Yes. So the -- I'm sure, as the original Legacy Horizon 3 was rolled out, that there were probably a lot of 4 comments on the usability of the system. It looks like 5 they took the opportunity to simplify the number of 6 screens that were required to manoeuvre through to do 7 the same functions that the Legacy Horizon system did, 8 in many more steps. 9 Q. Then, lastly, in order to simplify business processes. 10 Can you explain that, please? 11 A. Yes, so, you know, in addition to just simplifying the 12 screens, that implies a simplification of the processes 13 behind the screens, but it also includes, you know, 14 items such as reducing the number of reports that were 15 available to the SPMs at the branches. 16 Q. With that introduction in mind, can we turn to 17 component A, which is, again, the counter and the 18 peripherals. Was it right that the hardware components 19 for Horizon Online were almost identical to those for 20 Legacy Horizon? 21 A. Yes. 22 Q. However, an important change was that every branch 23 received a new router, a piece of hardware which allowed 24 the branch to connect to the data centre? 25 A. Yes. 127 1 Q. Did that router have a physical line connecting it to 2 the data centre? 3 A. Yes. 4 Q. But -- 5 A. It could have been, possibly -- it was either a fixed or 6 a mobile line but it was, effectively, a persistent 7 communication link to the data centre. 8 Q. You describe in your report that the new architecture 9 under Horizon Online had four layers: presentation, 10 interaction, business and services. Could we look at 11 those, please, in 4.6.5 of your report? 12 A. Yes. 13 Q. Can you explain what the presentation layer was? 14 A. So the presentation layer is simply the way the user 15 sees the screen and how the user interacts with the 16 system. 17 Q. The interaction layer? 18 A. That was the underpinnings of the presentation layer, 19 for instance, you know, how the menus were organised, 20 and it was also -- it also replaced -- so now that we're 21 online and now that -- and we'll get to this in 22 a second -- we don't need the Riposte messaging 23 technology in exactly -- we don't need that exact 24 mechanism to deliver the data now. It also will help to 25 replace how the transactions were communicated to the 128 1 campuses. 2 Q. Thirdly, you say that the business layer provided the 3 business applications in an object-oriented manner. 4 What do you mean by that, an "object-oriented manner"? 5 A. So "object-oriented manner" is a programming term and it 6 has to do with more efficiently organising your code for 7 repeatability and it's a style of programming. I think 8 that would be sufficient for explanation right now. 9 Q. Then, lastly, the layer -- the services layer. Can you 10 explain what that was, please? 11 A. Yes. This is -- so the service layer is, you know, the 12 set of objects that support all the applications and the 13 processing engine. It basically set the sequence of 14 processing steps for the counter to deliver its 15 services. 16 Q. Under Horizon Online, is it right that customer 17 transactions were not stored at the counter level? 18 A. That's correct. 19 Q. At the counter level, data including reference data, 20 process definitions and report definitions were stored 21 at the counter? 22 A. Yes. 23 Q. You say the service layer is the only layer of 24 communication with the data centre. Can you explain 25 that, please? 129 1 A. Um, it's -- the service layer, that was -- all the 2 functionality for communication resides in the service 3 layer and that's simply what I'm trying to communicate 4 there. 5 Q. The figure at the top of this page, at 4.11, you depict 6 the counter setup in a multi-counter branch in Horizon 7 Online, is that right? 8 A. Yes. 9 Q. Can you just talk us through that please? 10 A. Certainly. So if you remember from a multi-counter 11 example from the Legacy Horizon, the configuration 12 required one of the counters to be the Gateway PC, which 13 was responsible for communication directly to the data 14 centres or to the campuses. All of the other counters 15 basically were connected to that Gateway PC. The 16 difference here is that the counter, each one of the 17 counters are connected through the communication -- the 18 router hardware, which directs the transmission of the 19 transactions to the campuses. 20 So, really, what this is showing is that each one of 21 the counters can operate independent of the other 22 counters and, as we talked about, there is no 23 requirement or, in fact, it's not allowed for the 24 messages -- the transactions to be stored in messages on 25 the counters. The messages are communicated directly to 130 1 the campuses because it's a persistent online connection 2 and there's no need to worry about having them persist 3 at the counter level. 4 Q. Thank you. You say that the biggest change for 5 a subpostmaster would have been to the user interface 6 which changed significantly, and I think we can see that 7 over the page in figure 4.12 at the top of the page. If 8 we can just highlight those two diagrams next to each 9 other, those two screen prints. At the top of the page, 10 please. Thank you. 11 A. So you can see the look and feel is quite different and, 12 as I described a few minutes ago, the actual procedure 13 through the screens also was different based off -- you 14 know, produced a more efficient set of business 15 processes that were represented by a different set of 16 screens. 17 Q. Additionally, I think you tell us that there were some 18 changes to the available functionality, for example the 19 ability to transfer sessions between counters was 20 removed when Horizon Online was introduced; is that 21 right? 22 A. That's correct. 23 Q. That table can come down. Thank you. Horizon Online 24 was coded using java. That's different, isn't it, 25 because it replaced the Visual Basic components that had 131 1 been used under regular Horizon? 2 A. Yes. 3 Q. Skipping over component B, then, and going straight to 4 component C. You tell us that because Horizon Online 5 only stored transaction data at the data centre and not 6 at the counter, as you've explained already, the Riposte 7 message centre was no longer required? 8 A. Yes. 9 Q. Reference data is, however, still being stored locally 10 on the counter; is that right? 11 A. Yes. 12 Q. There was a change in messaging system using XML instead 13 of Attribute Grammar -- 14 A. Yes. 15 Q. -- is that right? Do you know why that was; can you 16 tell from the documentation? 17 A. It was -- I am going to speculate but I'm going to say 18 that the XML and Attribute Grammar were different 19 versions of the same concept and XML was probably more 20 universally accepted method of markup language as 21 opposed to the Attribute Grammar and they just took this 22 opportunity to switch. But I don't know that for sure, 23 but that's what I suspect. 24 Q. Then, lastly, you say a difference is that Horizon 25 Online used the TCP/IP protocol instead of UDP/IP to 132 1 send data between the counter and the data centre. Can 2 you explain that, please? 3 A. Yes, it's just more of an updated communication protocol 4 that they used. 5 Q. Then, lastly, in terms of the component parts of Horizon 6 Online, component D. You say there were various updates 7 made to the data centre. You highlight two: the branch 8 access layer and the branch database being new elements 9 introduced in Horizon Online. Can you just explain each 10 of those, please? 11 A. Certainly. So since the -- there was a switch in how 12 the transactions were stored and communicated to the 13 data centres, meaning that there's now a direct 14 communication of the transactions to the data centre, 15 the branch access layer was responsible for facilitating 16 that communication and the branch database was 17 essentially the management of that information after the 18 branch access layer received the information at the 19 campus. 20 Q. Thank you. 21 There are a number of updates to the Horizon Online 22 system. For our purposes, the most significant was in 23 2017 -- is this right -- as you've mentioned already: 24 the release of HNG-A Horizon Online Anywhere? 25 A. Yes. 133 1 Q. You say that you haven't been provided with any 2 substantial documents which detailed the changes 3 delivered, however you understand that one of the major 4 changes was that the operating system was upgraded from 5 Windows NT to Windows 10. That was necessary because of 6 the obsolescence of Windows NT? 7 A. Yes. 8 Q. Thank you. 9 Sir, I wonder whether we could take the afternoon 10 break there and come back at maybe 3.10? 11 SIR WYN WILLIAMS: Certainly. 12 MR BEER: If you're willing to sit a little later than usual 13 today, sir, we could finish Mr Cipione's evidence today. 14 SIR WYN WILLIAMS: Well, I'm perfectly happy to do whatever 15 is necessary to accommodate Mr Cipione and I'm sure he'd 16 much prefer to finish than come back tomorrow. 17 MR BEER: Thank you very much, sir. 18 (2.56 pm) 19 (A short break) 20 (3.10 pm) 21 MR BEER: Good afternoon again, sir. Can you see and hear 22 me. 23 SIR WYN WILLIAMS: Yes, I can, thank you. 24 MR BEER: Thank you very much, sir. 25 Can we turn, Mr Cipione, to balancing and rolling 134 1 over. You tell us in paragraph 4.7.1 of your report 2 about a process on Horizon which was prominent, which 3 you see referenced in the PinICLs, PEAKs and KELs, and 4 that is the concept of a cash accounting period. Can 5 you just tell us what a cash accounting period is, 6 please? 7 A. Yes, the cash accounting period was a weekly cycle -- 8 I believe it started Thursday and ended on the 9 subsequent Wednesday evening -- that was basically 10 a period of time at which point the -- a reconciliation 11 needed to be done. All the books and records needed to 12 be checked to make sure they were in alignment with the 13 Horizon System to the physical stock and cash that was 14 at the branch. 15 Q. You say that in 2004 you understand that Post Office 16 Counters Limited moved to a monthly trading period, a TP 17 cycle, with months made up of four or five weeks; is 18 that right? 19 A. Yes. 20 Q. The cash accounting periods, the CAPs, were sequentially 21 numbered one, two, three, four, five, and mirrored the 22 financial year, starting in March each year and running 23 to 52 or 53 weeks later? 24 A. Yes. 25 Q. You say that the cash accounting period is of particular 135 1 interest as it acted, as you've just said, as a weekly 2 reconciliation point for a branch? 3 A. Yes. 4 Q. You say that the data stored in Horizon was compared 5 with the cash and stock physically held in the branch at 6 the end of the cash accounting period and that this 7 weekly reconciliation process was referred to as 8 "balancing"? 9 A. Yes. 10 Q. You say that this was undertaken for each stock unit in 11 the branch? 12 A. Yes. 13 Q. Only once subpostmasters had balanced all of their stock 14 units were they permitted to roll over the branch until 15 the next cash accounting period? 16 A. Yes. 17 Q. That process of balancing the stock units and moving to 18 the next cash accounting period were commonly referred 19 to as "rolling over" or a "rollover". 20 A. Yes. 21 Q. You explained that a cash accounting period is further 22 subdivided into balance periods. Can you explain those, 23 please? 24 A. Yes, a balance period was -- could be shorter than the 25 cash accounting period. It didn't have to be but it 136 1 could have been shorter. It was just a mechanism that 2 allowed the SPM, the subpostmasters and 3 subpostmistresses to balance incrementally on an interim 4 basis before the end of the cash accounting period. 5 Q. You tell us that the cash accounting period and the 6 balancing period were each displayed on the Legacy 7 Horizon screen and the trading period and the balancing 8 period were displayed on the Horizon Online screen? 9 A. Yes. 10 Q. Can we just look at those, please, go back to page 48. 11 Look at page 48, please, and look at the table at the 12 top of 48, the pair, side by side. If you can highlight 13 those. Thank you. 14 So you tell us that the cash accounting period and 15 balancing period were displayed on the left-hand screen, 16 which is the Legacy Horizon screen. Could you point out 17 where they are on there? 18 A. Yes, I'm afraid that we must have switched out the 19 graphic on this -- on the Legacy Horizon system because 20 it doesn't show that particular information but, on the 21 Horizon Online, you can see the balance period and the 22 trading period on the bottom. But it was inadvertently 23 removed from this particular graphic. 24 Q. Do you see on the left-hand screenshot -- 25 A. Oh, I'm sorry. 137 1 Q. -- to the right of the line, two lines from the bottom 2 in white it says, "TP:07/BP:01", I think. 3 A. Yes, that is correct, I apologise, I think my reference 4 in the report was back to a prior graph, in which case 5 it wasn't there, so I apologise. 6 Q. Exactly. That's why I went to this one rather than the 7 one you referenced in your report because I think when 8 you were screenshotting, that got cut -- 9 A. Yes. 10 Q. -- in the figure on 4.8 on page 37 but I think it is 11 actually shown on this one? 12 A. It is. 13 Q. Then on the right-hand, on Horizon Online, can you point 14 out where the TP and BP are shown on that, please? 15 A. On the bottom bar you can, right as the white shows up 16 above it, the "TP:01" and the "BP:03", if my eyes are 17 working correctly. 18 Q. Can I ask, is there a facility to use a pointer on this 19 display system. 20 Can you point, please, Frankie, to the TP and BP on 21 the line at the foot on the right-hand diagram. 22 Perfect. Then the BP? Thank you very much. 23 I hope that's visible to you, sir. 24 SIR WYN WILLIAMS: Yes, it is, and could we just do it with 25 the Legacy Horizon as well, just to be -- so that I'm 138 1 sure I'm looking at the right thing. 2 MR BEER: Left-hand diagram, thank you. 3 SIR WYN WILLIAMS: That's it. I've got it. That's fine. 4 MR BEER: Thank you very much. 5 We can go back to page 50 of the report and we're up 6 to paragraph 4.7.5, where you say that, in order to 7 balance and roll over, a subpostmaster had to undertake 8 various steps, a summary of which you reproduce. This 9 process would be undertaken for each stock unit that the 10 branch operated and you outlined five processes. Can 11 you just walk us through those, please? 12 A. So first, (a) check all the stocks held in branch 13 against the system held values and adjust the values in 14 the system where, so to the extent that there was 15 a difference, enter what was actually physically on 16 hand; declare stamps held in the branch; declare the 17 cash held in the branch; then produce a balance sheet -- 18 a balance snapshot report and complete the mandatory 19 checks, making adjustments to transactions and stock and 20 cash declarations where inconsistencies are identified, 21 or accepting any discrepancies that the Legacy Horizon 22 system identified between its calculated values and 23 those from the declarations; and then confirm in Horizon 24 that they wished to roll over this particular stock unit 25 to the next CAP. 139 1 Q. Thank you very much. You say in 4.7.6 that: 2 "Any loss or gain that was identified through this 3 process must be either posted to the Suspense Account 4 ... or had to be corrected by the [subpostmaster] adding 5 funds to the till (if a loss) or [taking money out] (if 6 a gain)." 7 A. Yes. 8 Q. You explain that the posting of discrepancies to the 9 suspense account was only made once the stock unit had 10 rolled over to the next cash accounting period. Can you 11 just explain that? 12 A. So as the -- as it's rolled over, so the discrepancy as 13 that was rolled over to the next counter unit would then 14 cause the transaction -- basically cause an update to 15 the suspense account. 16 Q. Thank you. 17 You explain that, once stock units had been balanced 18 and rolled over, a subpostmaster would produce the cash 19 account report for the branch, which would summarise the 20 position across all stock units and then the 21 subpostmaster would check this report and, if they were 22 happy with this, they would roll over the branch to the 23 next cash accounting period; is that right? 24 A. Yes. 25 Q. You explain in 4.7.9 that that which you have described 140 1 is intended to provide an overview of what steps were 2 followed as part of the rollover and provides the 3 context for the process of checking that receipts and 4 payments matched. 5 A. Yes. 6 Q. You say that the stock unit balancing process consists 7 of accumulating all of the receipts for the stock unit 8 concerned and all of the payments for that stock unit in 9 the period for which the report is being produced and 10 ensuring that the total value of receipts matches the 11 total value of any payments. When that state is reached 12 the stock unit is said to have balanced, yes? 13 A. Yes. 14 Q. You explained some definitions of payments and receipts, 15 which might be obvious but can you just talk us through 16 those, please, in the next paragraph? 17 A. Certainly. Payment is essentially a transaction that 18 was a payment to the customer in and a receipt is 19 a transaction that's a payment from the customer. 20 Q. Thank you. 21 You explain that it's not intuitive that payments 22 and receipts should match one another, however it's your 23 understanding that the balancing of payments and 24 receipts factored in the cash and stock balance at the 25 end (sic) of a cash accounting period, as well as the 141 1 cash and stock balance at the end of a cash accounting 2 period; is that right? 3 A. Yes, it takes into account the beginning balances and 4 the ending balances. 5 Q. You've set out -- and we needn't go through them -- at 6 4.7.14, the balance equations for a stock unit in six 7 examples. 8 A. Yes. 9 Q. You draw these threads together at paragraph 4.7.15 on 10 page 53, in an example table at table 4.4. Can you talk 11 us through this, please? 12 A. Certainly. This is a simple example. We start off with 13 cash -- a brought forward balance for both cash and 14 stock, which, in this case, just represents stamps. The 15 cash was £5,000 and the stock was £500. You can see 16 that the different receipt amounts relate to payment for 17 a TV Licence, £100; payment of road tax for £75; 18 Alliance & Leicester giro deposit, £150; the purchase of 19 First Class stamps for cash of £5; and additional money 20 that was REMed in from POCL of £100. 21 On the other side of the ledger, you see A&L giro 22 withdrawals of £50; a pension payment of £25; savings 23 withdrawals of £100; and the issuance of the stamp of £5 24 to a customer, which results in the carried forward 25 balance of cash, £5,255, and stock, which is £495. 142 1 What this report shows, or the way this report 2 should be read, is you can see that the -- everything on 3 the left side, which is the total receipts, adds up to 4 £5,930 and everything on the right, for the payments, 5 also adds up to £5,930. You'll notice that the only 6 stock transaction was the £5 of stamps for cash and 7 you'll notice that that is -- that the stock is being 8 reduced by £5 and the cash is being increased by £5 for 9 that particular transaction. So that's the only one 10 that really goes between the stock and the cash. 11 Q. So for this cash accounting period, from 14 through 15 12 and then into 16, the total for receipts, 5,930, matches 13 that for payments, 5,930. So the stock unit is balanced 14 and therefore could be rolled over without the need for 15 adjustment or any further action? 16 A. Yes. 17 Q. I think it's fair to say that this side-by-side analysis 18 here is not something that you were able to see when you 19 looked at the data that you were provided by the 20 Inquiry? 21 A. That is correct. 22 Q. Is that because the format of a stock unit balance 23 report that was produced by Horizon Online, the manner 24 in which it produced its stock reports, was different, 25 because the way it came out of the tally printer was 143 1 a sequential list rather than a more useful table like 2 this? 3 A. Yes. 4 Q. Thank you very much. Can we turn to our penultimate 5 topic, please: error logging and remediation on Horizon. 6 This starts at page 55 of your report. Thank you. 7 You say that as part of the incident management 8 process for Legacy Horizon, ICL utilised a Fujitsu 9 proprietary core management system for logging errors 10 and defects. What do you mean by a proprietary core 11 management system? 12 A. It was built by and for Fujitsu. 13 Q. This was known as PinICL? 14 A. Yes. 15 Q. You say it was created from another ICL programme with 16 changes made at the request of ICL Pathway; is that 17 right? 18 A. Yes. 19 Q. It was used between '96 and 2003, prior to the PEAK 20 system -- about which we're going to hear in a moment -- 21 being introduced; is that right? 22 A. Yes. 23 Q. You explain at 5.1.2 the various ways in which a PinICL 24 can be raised. Can you just explain those to us, 25 please? 144 1 A. Certainly. PinICL could have been raised through 2 internal testing or monitoring. So, you know, let's go 3 back to our testing process. You know, the quality 4 assurance could have raised a PinICL or, you know, the 5 development team could have raised a PinICL. It also 6 could have been raised from feedback from the user 7 community, either through the user acceptance testing 8 phase or from the deployment and, you know, maintenance 9 phase of the work -- of the Horizon cycle. 10 Q. Post Office Counters could raise incidents themselves -- 11 A. Yes, that is correct. 12 Q. -- ie if a subpostmaster fed incidents back to them, 13 they could raise a PinICL with Fujitsu? 14 A. Yes. 15 Q. You say, I think, lastly, that through the Horizon 16 System Helpdesk -- 17 A. Yes, so users -- users of the system would not have 18 direct access to the PinICL system. Rather, they would 19 have access to the helpdesk facility that was put 20 together by ICL Pathway. 21 Q. That was called PowerHelp, yes? 22 A. Yes, that is correct. The internal team -- so while the 23 end user wouldn't directly raise a PinICL, if it was 24 deemed appropriate by their level 1 and level 2 helpdesk 25 people at ICL Pathway, it could eventually become 145 1 an item in the PinICL system. 2 Q. You helpfully describe for us the lines of support that 3 were available, the first of which was the Horizon 4 System Helpdesk, HSH, which you describe as the first 5 line of support being responsible for recording the 6 details of an incident diagnosing the problem and 7 attempting to resolve an issue. If it was unable to do 8 so, it would be routed to a second level of support 9 called the System Management Centre or SMC; is that 10 right? 11 A. Yes. 12 Q. You say that the SMC would determine if the incident was 13 a software code problem. How would it do that? 14 A. Um, the SMC would invest -- you know, or would read the 15 description, make a determination based off their 16 training and that was, basically -- that was their job: 17 to determine whether it was a software problem or not. 18 Q. You say that if the problem was a known error, 19 ie something described in a KEL, a Known Error Log -- is 20 that right -- 21 A. Yes. 22 Q. -- they would see whether there was a work around that 23 was written into the KEL and presumably suggest that 24 workaround? 25 A. Yes, that is correct. 146 1 Q. If there wasn't a workaround, the System Management 2 Centre would ensure that there was no duplicate calls 3 for the same problem. Just explain that? 4 A. Right. So it's very possible that if there were 5 an issue with Horizon, there could be multiple calls 6 coming in about the same issue, and perhaps there was 7 a difference in timing when, you know, call 1 to call 10 8 came in. To the extent that a PinICL had already been 9 raised for a particular issue, there was no reason to 10 raise a second PinICL for that same issue. So they 11 would simply acknowledge that this call was related to 12 a PinICL -- to an issue that had already been raised, 13 and not raise a second issue. 14 Q. If there was no known error log or no known workaround 15 in a KEL, if there was no duplicate call or if this call 16 was attached to a duplicate call that was being 17 escalated and if the incident was identified to be 18 a software problem, was the call then rooted to the 19 System Support Centre, the SSC -- 20 A. Yes. 21 Q. -- the third level of support? 22 A. Yes. 23 Q. Was that third level support, the SSC, responsible for 24 resolving incidents promoted by the System Management 25 Centre? 147 1 A. Yes. 2 Q. That resolution was recorded or ought to have been 3 recorded in the PinICL system; is that right? 4 A. That is right. 5 Q. The maintenance of the PinICLs was the responsibility of 6 the System Support Centre through resolution and 7 closure, with communications back -- right back to the 8 beginning -- to PowerHelp; is that right? 9 A. That is correct. 10 Q. If additional evidence was required, that's to resolve 11 the issue, the third level of support, System Support 12 Centre, would be engaged to gather the evidence -- 13 sorry, the second level would be engaged? 14 A. Yes, the second level. Yes. 15 Q. In the event that the SSC, third level, needed 16 assistance from third-party vendors, would they escalate 17 calls to the fourth line support, which dealt with 18 technology from outside suppliers? 19 A. Yes. 20 Q. Can we turn, then, to PinICLs and PEAKs over the page, 21 please. Each incident, you tell us, logged in the 22 PinICL system is referred to as a "PinICL"; is that 23 right? 24 A. Yes. 25 Q. The only ostensible difference between a PEAK and 148 1 a PinICL is what? 2 A. The PEAK system -- there was a PEAK system that replaced 3 the PinICL system using updated technology. There was 4 a big difference in the format of how they were recorded 5 in the technology that underpinned them but, 6 essentially, they carried out the same function. 7 Q. You say: 8 "As new PPs ..." 9 That's PinICLs or PEAKs; is that right? 10 A. Yes. 11 Q. "... are logged by a Team Member they are assigned 12 a unique reference number." 13 Is that right? 14 A. Yes. 15 Q. Which sets out when it was open, the last update to it, 16 its open or closed status, a summary of the issue, and 17 the product group. We'll see one of these in a moment. 18 A. Yes. 19 Q. If available, information is captured relating to work 20 packages, fixes, other PinICLs or PEAKs for reference; 21 is that right? 22 A. Yes. 23 Q. The way that both PinICLs and PEAKs are set out is 24 chronological and begins with the team member describing 25 the issue, assigning a call priority, a call type, 149 1 an estimated completion date, and routing to a team 2 leader who would or should review the call, providing 3 approval or rejection, and then rooting the call back to 4 the relevant team member; is that right? 5 A. Yes. 6 Q. Can we deal with KELs, please? Separately ICL Pathway 7 and Fujitsu maintained a knowledge base of information 8 that included known issues in the IT system; is that 9 right? 10 A. Yes. 11 Q. That knowledge base was known as the Known Error Log or 12 KEL? 13 A. Yes. 14 Q. An individual entry is referred to as a KEL; is that 15 right? 16 A. Yes. 17 Q. They contain information on how to address or rectify 18 issues that have previously been identified within the 19 system? 20 A. Yes. 21 Q. The responsibility for maintenance of KELs rested with 22 third and fourth line support; is that right? 23 A. Yes. 24 Q. They are often, or sometimes, referred to during the 25 resolution of a PinICL or a PEAK? 150 1 A. Yes. 2 Q. They contain structured attributes including the type, 3 summary, the open or closure date, the status of the KEL 4 and visibility. What does "visibility" mean? 5 A. How broadly it's been seen by the team. 6 Q. By the team or by anyone other than the team? 7 A. Well, everyone had access to the KEL system within 8 ICL Pathway. 9 Q. They contain, in the body, information covering the 10 symptoms, problems, solutions and related evidence. 11 A. That is correct. 12 Q. Can we turn, lastly, then, to some of the materials 13 provided to you and look at some examples. By way of 14 primary materials -- and these were date limited up 15 until and including the rollout -- I think you were 16 provided with extracted IT incident tickets, PinICLs, 17 yes -- 18 A. Yes. 19 Q. -- from Fujitsu's PinICL system, yes? 20 A. Yes. 21 Q. Equivalent extracted tickets from the PEAK system; is 22 that right? 23 A. Yes. 24 Q. Two archived PinICL databases, yes? 25 A. Yes. 151 1 Q. I think we'll hear in a moment that, in fact, you didn't 2 analyse, in agreement with the Inquiry, to any extent 3 the archived databases that you were provided? 4 A. Yes. 5 Q. Extracted records from the KEL system -- is that 6 right -- so a series of KELs -- 7 A. Yes. 8 Q. -- and a collection of monthly reports; is that right? 9 A. That is correct. 10 Q. Can we start with the PinICLs and PEAKs, then. Can we 11 look, please, at page 167 of your report. You 12 explain -- this is an appendix to your report -- 13 an example PinICL; yes? 14 A. Yes. 15 Q. You explain in the rubric at the top that this is 16 an example PinICL with some of the challenges of 17 interpreting this highlighted as call-outs. Do you 18 mean, by "call-outs", the boxes on the side, which have 19 an arrow back to the underlying text? 20 A. That's what I'm referring to. 21 Q. Thank you. You say you deliberately selected this as 22 an example PinICL because it contains more fulsome 23 descriptions and many PinICLs are more challenging to 24 interpret? 25 A. Yes. 152 1 Q. You say, by way of note, that a PinICL was written for 2 internal tracking purposes by the team trying to 3 investigate and resolve the issues, not written in a way 4 to give a complete and accurate explanation of all 5 investigatory steps, such as that somebody reviewing 6 them 20 years later could fully understand what had 7 occurred? 8 A. That is correct. 9 Q. Can we blow up the PinICL in 18.6, please, so it takes 10 up more of the page. Thank you. 11 So PinICLs recorded in a standard format certain 12 information; is that right? 13 A. That is correct. 14 Q. It's gone back. Thank you. 15 There's a number at the top, PinICL Expor and then 16 PC0044570. Was that a reference number, a unique 17 reference number? 18 A. Yes. 19 Q. There then is a "Summary", can you see the word 20 "Summary"? Just underneath the reference. 21 A. Oh, yes, yes. 22 Q. Was that, or meant to be, a summary of what the issue 23 was? 24 A. Yes. 25 Q. You can see there's an opened date, this one at 4.48 on 153 1 3 May 2000, I think that is? 2 A. That is correct. 3 Q. Then the last update and status, you can see that it was 4 last updated at 5.21 on 6 July, and that its status was 5 closed? 6 A. Yes. 7 Q. The customer, do you know who that referred to? Not who 8 that gentleman was, but what the -- who the customer was 9 in that field? 10 A. Um, generally it was the person generating the call to 11 the helpdesk that -- if this got promoted -- if it came 12 through the helpdesk, you know, the customer that 13 reported it. 14 Q. Then on the right-hand side, the product group and the 15 products at fault. Was this meant to identify the two 16 things that are set out there, in this case the product 17 group was the EPOSS, the electronic point of sale 18 system, and the desktop and the product within that as 19 the cash account? 20 A. Yes. 21 Q. Would there be a similar short descriptor on other 22 PinICLs? 23 A. Theoretically, yes. I did not -- I do not think that 24 everything was filled in all the time but I don't 25 remember that exactly. But I do not believe that -- but 154 1 generally, it was filled in. 2 Q. Yes, I mean, you say that you've picked a fulsome one, 3 which has got all of the information that we might want 4 to see. 5 A. Yes. 6 Q. In the reference section, what was that for? Can you 7 see that, the box on the left-hand side, a third of the 8 way down the page? 9 A. So this had, you know, this had different extra 10 information to the extent that they were available. So 11 the copy from -- you can see that even above, in the 12 summary, it's indicating that this PinICL was a copy of 13 PinICL PC0044007 that's repeated here, with the 14 indicator that it was a copy from. The SSC KEL, that -- 15 that's the KEL -- a KEL reference that could be related 16 to this that's shown here. The other, I'd have to look 17 more closely and remember what -- 18 Q. What that code is? 19 A. Yes. 20 Q. Then to the right, a products table. What was the 21 purpose of the products table? 22 A. So this is actually repeating the information that we 23 went through also. The product group, "EPOSS and 24 DeskTop" and the product name "Cash Account", which was 25 shown in the header. 155 1 Q. Then the activities log, which takes up the rest of this 2 page and, in fact, I think if we looked at this one, 3 we'd find it went on for pages and pages and pages. We 4 can see this page 1 of 8 in the bottom right-hand 5 corner. 6 A. Yes. 7 Q. What was the activities log? 8 A. This is the running dialogue of the process of analysing 9 and hopefully remediating this particular issue in the 10 PinICL. 11 Q. It proceeds, is this right, in chronological order with 12 earliest entry first and latest entry last? 13 A. Yes. 14 Q. In some of the added boxes, the call-outs, you have 15 noted three things: firstly that on this one, there's 16 an example of some typos, shorthand and acronyms that 17 are used that you as a reviewer would have to decipher? 18 A. Yes. 19 Q. Can you just explain the significance or relevance of 20 the second call-out explanation? 21 A. So the current release version of the system, so this 22 indicates to me that the dialogue here believes that the 23 new release of Horizon, the CSR-C13R, should -- to me, 24 this indicates that this problem should be resolved when 25 this release of the software had been implemented. 156 1 Q. Then, lastly, you note an entry at 8.11 in the morning 2 on 4 May, "Call transferred multiple times within 3 a ticket". What was the significance of that? 4 A. This just to highlight the fact that you'll see the 5 username changing throughout the chronology of this 6 particular PinICL and we're just showing you where that 7 particular change was initiated. So, in this case, 8 Richard Coleman was assigning the PinICL to Garrett 9 Simpson. 10 Q. I think you were provided with 56,489 of these -- 11 A. Yes. 12 Q. -- as individual PDFs -- 13 A. Yes. 14 Q. -- in a date range of 7 July 1996 and 31 December 2000? 15 A. Yes. 16 Q. Thank you. Can we turn to PEAKs, please, and look at 17 an example PEAK over the page, at page 168. Again, if 18 we can just blow that up so it's a bit clearer. Thank 19 you. 20 So we know that in 2003, ICL replaced PinICLs with 21 the PEAK system. Is it right that the PinICL system was 22 archived and any open tickets from the PinICL system 23 were transferred/migrated across to the PEAK system? 24 A. That is correct. 25 Q. The purpose of the PEAK system served a similar function 157 1 to the PinICL system, and therefore, as you say in your 2 report, the origins of the tickets within it would be 3 much the same as those identified for the PinICL 4 system -- 5 A. Yes. 6 Q. -- ie the sources by which these things would be raised? 7 A. Yes. 8 SIR WYN WILLIAMS: Sorry, Mr Beer, I am a little lost. 9 We've gone over the page, as I understand it but, unless 10 I'm lost, this is still a PinICL. My example is of 11 a PEAK is on page -- 12 MR BEER: 172. Quite right, sir. 13 172, please. Thank you. 14 You were telling us that it performed a similar, if 15 not identical, function and, therefore, the origins, the 16 sources of it, would be the same? 17 A. Yes. 18 Q. Can we see similarities in layouts between a PEAK and 19 a PinICL? 20 A. We definitely can see similarities. Much of the tagging 21 information that you saw in the PinICLs exist in the 22 PEAKs and, importantly, the running dialogue is in the 23 body, in the activity body of the PEAK, just as it was 24 in the PinICL. 25 Q. But set out slightly differently? 158 1 A. I'm sorry? 2 Q. But set out slightly differently? 3 A. Yes, yes. The format is different. 4 Q. Can you just talk us through the four or five lines at 5 the head of the PEAK, and what information we're being 6 provided with there? 7 A. Certainly. So we see the call reference, a unique 8 identifier for this PEAK. The release for resolving 9 this PEAK appears to be targeted at CSR-C13_2R. The 10 call type that has been assigned to this PEAK is a live 11 incident. The group that is working at this particular 12 issue is the EDSC, with a target date resolving this of 13 19 May 2000. 14 Q. You make the point in the box at the top there that when 15 you were interpreting these PEAKs, as with the PinICLs, 16 there were acronyms that you had to look up to find out 17 what they meant? 18 A. Yes. 19 Q. Yes, and then on the right-hand side? 20 A. So on the right-hand side, the first call-out box talks 21 about -- this ticket is referring to another call, which 22 indicates that they've seen this issue before, and the 23 second call-out box is the ticket is advised to close, 24 then gets reassigned, but no detailed explanation. It's 25 a bit of confusion for me, as a reader, to, you know, 159 1 understand exactly what's going on based on this 2 description. 3 Q. I think you were provided with 13,442 -- I'm sorry, 4 16,530 PEAKs; is that right? 5 A. Yes. 6 Q. You found 13,442 duplicated PEAKs in the PinICL 7 population; is that right? 8 A. Yes. 9 Q. So what was to done as a result of that? 10 A. We picked the PinICLs to use for our processing. 11 Q. You say that another issue concerned you, and that was 12 the ordering of the activities table in the PinICLs; is 13 that right? 14 A. Yes. 15 Q. What was the difficulty there? 16 A. As the original population of PinICLs were delivered to 17 me and my team, it became apparent that the actual 18 chronology shown within the activities log was not in 19 actual chronological order, and it made it obviously 20 very difficult to read. 21 Q. If we can go over the page in your report, that's 22 page 60 we were at, thank you. At the foot of the page, 23 I mentioned letter that you didn't examine the PinICL 24 archive databases that were provided to you, in 25 agreement with the Inquiry. At the foot of the page, 160 1 you deal with the summary of the PinICL and PEAK data 2 used to undertake your review, and you say that this 3 dataset changed over the course of the review as you 4 receive multiple copies of the same or very similar data 5 across different deliveries. You had to make decisions 6 as to what datasets to use. 7 You also had two analysis workstreams which were at 8 different states of progression when some of the 9 additional data was provided. You decided that these 10 workstreams should, in some cases, use different 11 datasets. You describe the two workstreams as 12 "Analytics" and "Document Review". Can you explain what 13 you mean by those, please? 14 A. Certainly. So the analytics review was trying to -- was 15 using some of the data elements that we called out when 16 looking at the example, PEAKs and PinICLs. Not the 17 dialogue, necessarily, but some items that were captured 18 from within the dialogue, or within the header 19 information of those documents. Those were used simply 20 for grouping and summing and creating charts based off 21 of the information in those structured data components. 22 A document review workstream, on the other hand, 23 existed with the intent of identifying different groups 24 of individual PEAKs or PinICLs to be reviewed manually. 25 So in other words, we wanted to read everything -- we 161 1 wanted human beings to read everything, to gain as good 2 an understanding of the documents as possible. 3 Q. I think you summarised, in the table at 6.1, the 4 documents used in your review by reference to the series 5 of documents that you received and the use to which they 6 were put; is that right? 7 A. Yes. 8 Q. So if you start in the middle column, the reference and 9 description, the A1, A2, A3, A4, A5, B1, B2, are the 10 series of documents that you received -- 11 A. Yes. 12 Q. -- in the course of your review. The total documents 13 provided the use to which they were put? 14 A. Yes. 15 Q. You say that neither the analytics nor the document 16 review workstreams were used in relation to the first 17 dataset, A1, because it was agreed with Fujitsu that 18 these data contained errors and were therefore replaced 19 by documents at A4 -- is that right -- 20 A. Yes. 21 Q. -- and A2, as it was recommended by Fujitsu that you use 22 the PEAK versions of these, which were contained in the 23 dataset B1; is that right? 24 A. Yes, that is right. 25 Q. If you turn over the page, please, to paragraph 6.2.25, 162 1 can you explain 6.2.25, please? Can you explain the 2 discovery that you refer to there, please? 3 A. So this is simply explaining that while the PEAK system 4 did replace the PinICL system, the archive databases 5 that were received contained all of the information that 6 was in the PEAK system. So it was appropriate to use 7 these archive databases for purposes of our analytics 8 workstream. 9 Q. Thank you. Can we turn to Known Error Logs, please, and 10 can we turn up page 174 of your report, please. Is this 11 is an example of a KEL, a Known Error Log? 12 A. Yes. 13 Q. You explain that this was a knowledge management tool 14 used by both ICL Pathway Limited and Fujitsu to explain 15 how to deal with, or to work around, issues that arose 16 in relation to the Horizon System. Can you talk us 17 through it, please, using this as an example? 18 A. Yes, certainly. So to the extent that an issue -- that 19 Pathway was aware of an issue, they had this mechanism 20 to record, you know, the -- you know, some general 21 information which is contained in the header, you know, 22 that -- the type, the title, the summary. You know, key 23 release dates, keywords that are associated with it. 24 It also describes the symptoms that describe the 25 particular issue, as well as what they believe the 163 1 problem is. Importantly, they also provide some text 2 for what the helpdesk should communicate back to any 3 user reporting a similar problem. So in this case, the 4 solution that the helpdesk should recommend to the user 5 is to reboot the counter and, if the message reappears, 6 then send to an engineer. 7 So what -- the purpose of a KEL is, you know, to 8 document known errors, but to also provide remedial, or 9 recommendations to users reporting similar problems. 10 Q. In terms of the header, "HORIZON KEL rcoleman3549n", in 11 your report, you say this contains the metadata for the 12 KEL, and it included the name of the Fujitsu employee 13 who raised it, the identifier for the PEAK or PinICL 14 that originated the KEL and the version number of the 15 KEL; is that right? 16 A. That is correct. 17 Q. That's all in that top line, "HORIZON KEL"? 18 A. Yes, you can see that the PEAK information is -- the 19 PEAK identifier is the bit below. 20 Q. The symptoms, is this a fairly typical description of 21 symptoms that we see here? 22 A. Typical, in terms of -- 23 Q. In terms of length and detail? 24 A. Yes. 25 Q. Is this a description from the perspective of the 164 1 reporting person, rather than the person to whom it has 2 been reported? 3 A. On reading this, this looks like a Horizon-centric 4 description of the problem. 5 Q. Not something that the subpostmaster might -- 6 A. No, no. 7 Q. So you've explained what the problem and the solution 8 was, or what those parts of the KEL were intended to 9 achieve. In terms of distribution or availability, 10 viewability of the KEL, is it your understanding that 11 the KEL system was available in read-only mode to the 12 first line support, that's the Horizon Helpdesk? 13 A. Yes. 14 Q. The second line support, the System Management Centre? 15 A. Yes. 16 Q. The third line support, the System Support Centre? 17 A. Yes. 18 Q. That third line support, as well as having read-only 19 access, was permitted to create a new KEL themselves or 20 to make changes to an existing KEL, to update a KEL? 21 A. That is correct. 22 Q. You say in your report that you noted that not all 23 sections of a KEL were completed in all of the KELs, 24 which indicated to you that not all of the sections were 25 mandatory when either creating or updating a KEL; is 165 1 that right? 2 A. Yes. 3 Q. Is it right that it's your understanding that the KEL 4 system, the Known Error Log system, was replaced in 5 about July 2009 by Knowledge Base or KB, a new system? 6 A. Yes. 7 Q. You had, I think, 656 KELs produced to you in HTM 8 format; is that right? 9 A. In HTM, yes. 10 Q. They were between the dates of 26 May 1998 and 11 28 December 2000 -- 12 A. Yes. 13 Q. -- and is the last species of principal documentary 14 information that's provided to you a management report? 15 A. Yes. 16 Q. You had 105 of those given to you, and I think they 17 included 19 Pathway Programme monthly reports 18 summarising the business development activities of the 19 Pathway Programme -- 20 A. Yes. 21 Q. -- 13 monthly joint implementation reports? 22 A. Yes. 23 Q. They're jointly issued by ICL Pathway and Post Office 24 Counters Limited? 25 A. Yes. 166 1 Q. All Pathway customer service reports containing 2 summaries of the performance of the ICL Pathway Customer 3 Service Business Support Unit? 4 A. Yes. 5 Q. Then the largest category, 44 Pathway monthly reports, 6 being a comprehensive management report for ICL Pathway 7 ranging from October '96 to December 2000? 8 A. Yes. 9 Q. Plainly, you didn't get every report for every month in 10 that time range? 11 A. That is correct. 12 Q. The two managing directors were the approval authorities 13 for those reports, JH Bennett, Mr Bennett, was the 14 approval authority up until November 1999, and M Stares. 15 Mr Stares was the approval authority beginning in 16 January 2000? 17 A. Yes. 18 Q. They followed a typical format; is that fair to say? 19 A. Yes. 20 Q. You set out the ten parts of them in Roman numerals in 21 paragraph 6.4.1(i) to (x) of your report? 22 A. Yes. 23 Q. You focused on that largest category, the 44 reports, 24 the Pathway monthly reports? 25 A. Yes. 167 1 Q. Four of them were duplicative, and therefore you had 2 40 reports to analyse? 3 A. That is correct. 4 MR BEER: Mr Cipione, they're the only questions that I ask 5 of you. I've had a question fed through to me. If you 6 just give me one moment. (Pause) 7 In fact I'm not going to ask that question now. 8 I think that's for next time. 9 A. Okay. 10 MR BEER: They're the only questions I ask of you at the 11 moment. You're returning to us, I think, on 18 and 12 19 November. Ordinarily, the chairman would say to you 13 that you're in purdah, that you shouldn't discuss your 14 evidence with anyone, or the evidence that you are to 15 give, with anyone. 16 Can I ask for permission, sir, to be released from 17 that embargo and so that Counsel to the Inquiry can 18 speak with Mr Cipione in the intervening period, 19 including about his evidence? 20 SIR WYN WILLIAMS: I'm very much inclined to accede to that 21 request. I note that there are a number of CPs in the 22 room. If anyone wishes to oppose that suggestion, now 23 is the time, otherwise I'm going to allow Mr Beer to do 24 as he suggests. 25 MR BEER: Thank you very much. 168 1 Sir, we said to Core Participants that now, on this 2 occasion, might not be the opportunity to ask Mr Cipione 3 questions, that they may wish to hold those over until 4 the November sessions. I think all Core Participants 5 have done so, in that we haven't received any questions 6 that other people wish to be asked on this occasion. 7 So, sir, unless you had any questions of Mr Cipione, 8 that's the end of his evidence for today. 9 SIR WYN WILLIAMS: No, I don't have any questions, 10 Mr Cipione, and I would just like to thank you for the 11 clarity of your written and oral evidence to date, and 12 look forward to seeing you again in a few weeks' time. 13 THE WITNESS: Thank you, sir. 14 MR BEER: Sir, thank you very much. That means that we're 15 not sitting tomorrow. We won't interfere with the order 16 of witnesses because they've all been warned for 17 specific periods of time over the coming months, and 18 therefore we sit again at 10.00 am on Thursday. 19 SIR WYN WILLIAMS: No, I agree with that. I think we should 20 stick to the timetable unless we're beginning to lose 21 time. When we're gaining time, I think we can use it 22 for other useful purposes, Mr Beer. 23 MR BEER: Thank you very much, sir. Can I thank you also 24 for spotting the deliberate error earlier on when I -- 25 SIR WYN WILLIAMS: I think that was a test to ensure that I 169 1 wasn't too remote from the proceedings, Mr Beer. 2 MR BEER: Very much so. You got me, sir. Thank you. So 3 10.00 am on Thursday. 4 SIR WYN WILLIAMS: Yes. Well, actually, since it's highly 5 likely that I will appear in person on Thursday, could 6 we just say 10.15 so as to avoid any possibility that 7 people will have to wait for me? 8 MR BEER: Sir, thank you very much. Yes. 9 (4.08 pm) 10 (The hearing adjourned until 10.15 am on Thursday, 11 20 October 2022) 12 13 14 15 16 17 18 19 20 21 22 23 24 25 170 1 I N D E X 2 CHARLES CIPIONE (sworn) ..............................1 3 Examined by MR BEER ...........................1 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 171