1 00:00:00,017 --> 00:00:02,497 Funded by Snaps Everywhere. Thank you. 2 00:00:06,957 --> 00:00:10,237 Thank you. Uh, so I'm Kevin Hortense. 3 00:00:10,357 --> 00:00:17,097 I've been around the community for 20 years or so, uh, and I'm working for Inukia Haute Couture. 4 00:00:17,417 --> 00:00:21,617 Uh, as part of my involvement with Inukia Haute Couture, we ended up working 5 00:00:21,617 --> 00:00:27,497 with Canonical on what we are calling KD Neon Core, and that's what this is 6 00:00:27,497 --> 00:00:30,517 about. So, you might have heard about Ubuntu Core. 7 00:00:30,837 --> 00:00:33,957 The definition they give is basically Ubuntu Core is a minimal, 8 00:00:34,097 --> 00:00:36,917 secure, and strictly confined operating system. 9 00:00:37,957 --> 00:00:41,457 It's been designed for embedded systems initially. 10 00:00:41,617 --> 00:00:48,077 It's basically the emu-table version of Ubuntu, just like the equivalents on 11 00:00:48,077 --> 00:00:52,297 the Red Hat side, OpenSUSE side, and so on. Supposed to be secure by design. 12 00:00:53,017 --> 00:00:58,017 Everything is containerized in there. and all the components are in Snaps. 13 00:00:58,657 --> 00:01:02,597 Interestingly on that one, though, even the kernel or the SnapD, 14 00:01:02,777 --> 00:01:06,097 which is what orchestrates the Snaps, they're also inside of Snaps, right? 15 00:01:06,157 --> 00:01:08,957 So everything is containerized. 16 00:01:09,877 --> 00:01:16,817 That makes it easier for the over-the-air updates and there's an automated rollback system if needed. 17 00:01:17,237 --> 00:01:22,357 Then I came up with the idea of, let's extend that and go toward Ubuntu Core 18 00:01:22,357 --> 00:01:26,817 Desktop. So, the idea is to have a fully containerized desktop this time, 19 00:01:26,917 --> 00:01:29,697 where each component is immutable and isolated. 20 00:01:30,237 --> 00:01:32,117 So, it's been announced last year, 21 00:01:32,951 --> 00:01:36,891 And they made a full desktop session on top of Ubuntu Core. 22 00:01:37,871 --> 00:01:41,551 The idea is to have the same benefits that Ubuntu Core itself, 23 00:01:41,951 --> 00:01:48,091 but for user-facing GUIs, to make it harder for malicious software to get there 24 00:01:48,091 --> 00:01:54,751 or to spread, to not have updates which leave you in an unstable state, 25 00:01:55,391 --> 00:02:02,591 to make it all easier to audit, verify, and also to avoid having inconsistencies from system to system. 26 00:02:02,751 --> 00:02:07,131 They are supposed to be all identical as long as they have the same components. 27 00:02:07,811 --> 00:02:11,131 That comes with extra challenge in that particular space, though, 28 00:02:11,851 --> 00:02:14,731 because the constraints are not exactly the same as for embedded systems. 29 00:02:15,911 --> 00:02:20,371 It's actually harder to draw the boundaries, because all desktops are kind of 30 00:02:20,371 --> 00:02:23,191 like a big blog, right? So it's harder 31 00:02:23,191 --> 00:02:26,371 to draw the boundaries to actually containerize properly everything. 32 00:02:27,491 --> 00:02:31,951 And also with the rollback system and everything, since the desktop are big, 33 00:02:32,011 --> 00:02:35,811 we're generally using way more storage than for an embedded system. 34 00:02:37,551 --> 00:02:42,491 Next to that, there's been this idea floating around of KDE Neon Core. 35 00:02:43,371 --> 00:02:47,851 And the idea, in a nutshell, that's all the KDE Neon benefits but on top of 36 00:02:47,851 --> 00:02:50,951 Ubuntu Core, which changed the base instead of a traditional Ubuntu. 37 00:02:51,391 --> 00:02:55,471 We're following a similar architecture to the Ubuntu core desktop, 38 00:02:55,791 --> 00:02:59,371 except that this time that's a Plasma-based user experience, 39 00:02:59,611 --> 00:03:04,651 and we're trying to provide the greatest and latest of KDE software. 40 00:03:05,151 --> 00:03:11,131 It also provides, in that effort, we're also trying to build up and provide 41 00:03:11,131 --> 00:03:14,731 building blocks to make it easier to snap the KDE applications, 42 00:03:15,971 --> 00:03:18,431 even for use outside of Ubuntu Core. 43 00:03:20,171 --> 00:03:25,311 All right. This will be kind of a technical talk, so I will get back a bit more 44 00:03:25,311 --> 00:03:29,031 into the basics of the confinement of Snap, starting with how do you make a Snap. 45 00:03:30,131 --> 00:03:33,511 That requires a build recipe so that there is snapcraft.yaml, 46 00:03:34,151 --> 00:03:38,311 and then you have a tool, snapcraft, build the package, end of story, 47 00:03:38,391 --> 00:03:40,731 basically, right? You still have to publish and so on. 48 00:03:41,491 --> 00:03:44,911 In the recipe, you find metadata to declare your application, 49 00:03:45,151 --> 00:03:49,131 the targeted base system, because you build on some, you expect some particular 50 00:03:49,131 --> 00:03:52,631 runtime to be able to run what you're building. 51 00:03:53,051 --> 00:03:56,671 And you declare all the apps which are provided by the package. 52 00:03:56,851 --> 00:04:00,631 You can have more than one app inside of a package, and that's valid. 53 00:04:01,071 --> 00:04:04,531 You also declare the interfaces which are used for all of those, 54 00:04:04,771 --> 00:04:09,091 so which can be on the plug side or the slot side, depending if you're consuming or providing. 55 00:04:10,931 --> 00:04:17,171 And then there's the actual build recipe, so you declare which packages you 56 00:04:17,171 --> 00:04:23,531 need to be able to build, you declare how to actually build each of the parts which are in your snap. 57 00:04:23,871 --> 00:04:26,071 So that's this kind of YAML. 58 00:04:26,965 --> 00:04:29,585 You, so I'm too small, I cannot jump to point. 59 00:04:31,885 --> 00:04:37,025 So you find, so that's basically the metadata, right? And the base system that you're targeting. 60 00:04:38,085 --> 00:04:41,725 And then we see the plugs here. We see some slots here. Okay, 61 00:04:41,785 --> 00:04:44,425 so that this kind of very simple declaration I'm talking about. 62 00:04:44,685 --> 00:04:49,605 And in the case of KDE apps, we have this one, which is important for us. 63 00:04:49,665 --> 00:04:53,785 That's one of the things which makes it easier for KDE apps because lots of 64 00:04:53,785 --> 00:04:57,545 our apps have similar expectations of what they need. 65 00:04:58,045 --> 00:05:03,185 So that's factored in what they call an extension, so that the KDE Neon 6 extension, 66 00:05:03,445 --> 00:05:06,465 which pre-wires some of what's needed. 67 00:05:07,265 --> 00:05:12,385 What's pre-wired? That's mainly some environment variables. So we position the path properly. 68 00:05:12,725 --> 00:05:16,345 The XTG underscore yada yada are positioned and so on. 69 00:05:17,205 --> 00:05:22,405 We also declare the build time, run time dependencies on KDE frameworks and 70 00:05:22,405 --> 00:05:24,885 Qt and some common plugs. 71 00:05:25,185 --> 00:05:29,905 We're providing a desktop application so we need to be able to connect to the actual desktop. 72 00:05:30,185 --> 00:05:34,465 We're using OpenGL so we need that and so on. So a few of those. 73 00:05:36,325 --> 00:05:40,945 Once you have a package, how does it work actually when you run it? 74 00:05:41,325 --> 00:05:45,845 That's two steps when you run a snap. 75 00:05:46,825 --> 00:05:52,345 When an application is launched there's first snap confine which is which is 76 00:05:52,345 --> 00:05:58,265 executed, and this one we set up the environment, the execution environment, sorry. 77 00:05:59,490 --> 00:06:03,250 This one will position environment variables like the homeDate of the user, 78 00:06:03,530 --> 00:06:05,650 some snap specific variables and so on. 79 00:06:06,190 --> 00:06:11,470 Then it will start to do funky stuff like providing a private namespace for 80 00:06:11,470 --> 00:06:14,330 mount points, providing a private TMP. 81 00:06:14,550 --> 00:06:17,250 So you have no way to actually see what's going on with the other ones. 82 00:06:17,470 --> 00:06:23,170 And then it will start setting up SecCom filters, APARMOR profiles to restrict 83 00:06:23,170 --> 00:06:27,470 what the application can do and the kernel, which basically arbiters all of that. 84 00:06:27,470 --> 00:06:32,170 Once it did that, there's the actual execution environment you want for your 85 00:06:32,170 --> 00:06:34,590 app. And then it passes to SnapExec. 86 00:06:34,770 --> 00:06:40,430 SnapExec, what it does is looking in the metadata file which has been generated 87 00:06:40,430 --> 00:06:45,890 with the snap, find out what is the actual command for the application and just run that. 88 00:06:46,050 --> 00:06:49,190 But at that point, you're inside of the confined environment. 89 00:06:51,150 --> 00:06:54,090 Applications on SnapCraft, they can also be declared as daemons. 90 00:06:54,090 --> 00:06:58,790 If you do that, you also get extra files next to that which are generative, 91 00:06:58,870 --> 00:07:02,270 which are basically systemd service files. 92 00:07:03,210 --> 00:07:07,830 Except that those ones don't run the command directly. They go through snap to run the command. 93 00:07:10,330 --> 00:07:15,930 At that point, one has to wonder where are the filters for seccomp and the profiles 94 00:07:15,930 --> 00:07:17,450 for our power mark coming from. 95 00:07:17,790 --> 00:07:22,350 That's snapd which basically manages this. When you install and uninstall some 96 00:07:22,350 --> 00:07:26,110 things, it would create or remove the profiles. 97 00:07:26,610 --> 00:07:29,910 And then when you connect, disconnect some of the interfaces, 98 00:07:30,110 --> 00:07:33,210 it will update said profiles following some rules, 99 00:07:33,430 --> 00:07:36,770 which means that somewhere in Snapd, that will be important later, 100 00:07:36,970 --> 00:07:42,850 there is code which basically maps from those nice desktop and OpenGL names 101 00:07:42,850 --> 00:07:45,090 that I showed before to actual templates. 102 00:07:47,814 --> 00:07:51,154 All right, the architecture for KD Neon Core is the following currently. 103 00:07:52,054 --> 00:07:57,254 So everything which is blue, that's components that we've been working on for that project. 104 00:07:57,454 --> 00:08:01,034 Everything which is orange, that's stuff that we've been reusing as is, 105 00:08:01,114 --> 00:08:06,094 and which is on a regular Ubuntu Core or Ubuntu Core desktop. 106 00:08:06,954 --> 00:08:11,114 There are several things here. So the easiest thing to spot that's at the top, 107 00:08:11,154 --> 00:08:12,594 the applications we've been working on. 108 00:08:12,694 --> 00:08:15,814 So there's a few more, but we've been working on a few applications. 109 00:08:15,814 --> 00:08:21,514 All of those, they depend on what they call a content snap, which is the one 110 00:08:21,514 --> 00:08:24,894 providing the Qt and KDE frameworks runtime. 111 00:08:25,694 --> 00:08:30,454 So all of those, they basically see the files from that content snap to just be able to run. 112 00:08:32,634 --> 00:08:39,554 But they're running inside of a session. And so they need to talk to something 113 00:08:39,554 --> 00:08:41,014 which provides the right interfaces. 114 00:08:41,154 --> 00:08:44,554 So I mentioned the desktop interface earlier. earlier, this desktop interface 115 00:08:44,554 --> 00:08:48,874 is provided by our Plasma Desktop Session Snap, which interestingly, 116 00:08:49,214 --> 00:08:53,714 the way it's done currently in the architecture is 16 kilobytes or so, 117 00:08:53,814 --> 00:08:57,874 something very, very small, because that's just a tiny bunch of scripts which 118 00:08:57,874 --> 00:09:01,854 all point to what we have under here, which is the base snap. 119 00:09:02,094 --> 00:09:06,194 And that's where in the base snap currently we're basically putting all the 120 00:09:06,194 --> 00:09:10,514 binaries which are making your usual KDE session. 121 00:09:11,914 --> 00:09:15,874 There's some technicality with this, which forces us to, I won't get into the 122 00:09:15,874 --> 00:09:18,554 details here because I won't have the time, but that forces us to have a third 123 00:09:18,554 --> 00:09:23,254 snap, otherwise we cannot make a system which boots with that particular base 124 00:09:23,254 --> 00:09:25,154 snap, but I'm not getting the details there. 125 00:09:26,914 --> 00:09:30,674 One thing to spot here is that currently what we're doing is that the Plasma 126 00:09:30,674 --> 00:09:33,334 Core 22 desktop, as I said, has all the binaries. 127 00:09:33,474 --> 00:09:38,654 Right now we're populating this with the DBN packages coming from KDE Neon, 128 00:09:38,774 --> 00:09:43,754 so the actual session is coming from that what in the runtime and the applications 129 00:09:43,754 --> 00:09:49,894 at the top that's actually rebuilt for for each application and each content 130 00:09:49,894 --> 00:09:51,994 snap so those are from the sources. 131 00:09:54,215 --> 00:09:57,375 Once you have that, you can get to the system that you want to boot. 132 00:09:57,855 --> 00:10:01,375 So we're providing SDDM inside of the base SNAP. 133 00:10:02,115 --> 00:10:05,555 And then you need to do... You boot and you have no user yet, 134 00:10:05,615 --> 00:10:08,715 right? So you need to do the first boot provisioning. 135 00:10:11,035 --> 00:10:14,695 SDDM is run outside of the confinement, which is easier in that case, 136 00:10:14,755 --> 00:10:16,835 because then we can do pretty much whatever we want. 137 00:10:17,855 --> 00:10:21,015 And so on first boot, we check if there's a regular user which exists. 138 00:10:21,235 --> 00:10:28,775 If that's not the case, We sneakily override the SDDM config so that it auto-logins 139 00:10:28,775 --> 00:10:35,255 as root directly with a very minimal session, spawns the wizard, 140 00:10:35,635 --> 00:10:38,055 the orange block there. 141 00:10:38,235 --> 00:10:43,355 Spawn that wizard, get your information, create the user, restart SDDM, 142 00:10:43,415 --> 00:10:47,255 and then at that point you have a regular login process. 143 00:10:48,655 --> 00:10:53,175 Interestingly, we had to do that work manually. I found out during that project 144 00:10:53,175 --> 00:10:56,595 that GDM actually has that by default. 145 00:10:57,255 --> 00:11:01,275 That's a feature which is provided, but we had to kind of build it. 146 00:11:02,395 --> 00:11:05,355 Once you're there, you can get in, you can boot Plasma. 147 00:11:05,515 --> 00:11:10,475 So one of the things is actually starting it and currently, or by default, 148 00:11:10,615 --> 00:11:13,675 we are driving everything with systemd for the boot. 149 00:11:15,335 --> 00:11:22,815 So here in red, I'm showing all the parts, right? are the services which are talking to systemd. 150 00:11:23,435 --> 00:11:28,655 The green ones, that's basically a single line of calling to QDBus, 151 00:11:28,695 --> 00:11:30,035 but they are not doing much otherwise. 152 00:11:30,275 --> 00:11:34,095 They are just one shot and they don't stay. And the orange ones, 153 00:11:34,275 --> 00:11:37,755 that the ones not talking to systemd, but staying around a bit longer. 154 00:11:38,575 --> 00:11:44,315 So the first one to kickstart that is startplasma, which is not really a systemd service, 155 00:11:44,515 --> 00:11:47,455 but it talks to systemd to say, okay set up that target 156 00:11:47,455 --> 00:11:50,775 so that i we can have a proper desktop it 157 00:11:50,775 --> 00:11:53,595 also says things like here is the environment i want 158 00:11:53,595 --> 00:11:57,615 you to give to every services which are started okay so 159 00:11:57,615 --> 00:12:02,115 we are doing those calls to systemd uh for uh for surplus man then that will 160 00:12:02,115 --> 00:12:07,095 kick start the rest queen wayland similarly will force some of the environment 161 00:12:07,095 --> 00:12:10,275 because you need to know where is the whalen circuit for instance this kind 162 00:12:10,275 --> 00:12:15,255 of stuff um and that's pretty much it and through the dependencies, 163 00:12:15,435 --> 00:12:18,175 we have the order of what will be started. 164 00:12:19,695 --> 00:12:24,655 That's a very simplified, idealistic version of the thing, but that's roughly how it goes. 165 00:12:27,263 --> 00:12:32,663 So, that gets us to an unconfined startup, because first you want to try your 166 00:12:32,663 --> 00:12:34,883 things without the confinement really activated, right? 167 00:12:34,943 --> 00:12:37,683 You put everything in the snap and you try to boot that, saying, 168 00:12:37,863 --> 00:12:42,063 yeah, don't put the filters, the apparmor profile yet, that would be fine, 169 00:12:42,163 --> 00:12:43,383 I'm just testing things, right? 170 00:12:43,463 --> 00:12:46,463 That's not what you want in production, but as a developer that actually helps. 171 00:12:47,823 --> 00:12:52,003 And so, that should be super easy, right? We just declared the necessary to 172 00:12:52,003 --> 00:12:55,483 have start plasma exposed so that it could be started, to have the necessary 173 00:12:55,483 --> 00:12:58,843 to have the desktop portal implementation for us. 174 00:12:59,283 --> 00:13:01,763 We boot and we get the black screen. 175 00:13:03,183 --> 00:13:08,463 Not exactly what we expected. Looking into it, it turned out to be a nice deadlock 176 00:13:08,463 --> 00:13:12,183 between Queen Wayland and the XEG desktop portal KDE implementation. 177 00:13:13,603 --> 00:13:17,703 And as Queen Wayland is stuck, well, nothing can proceed, nothing can display, 178 00:13:18,003 --> 00:13:21,523 and that's why you have a black screen. So why that deadlock in the first place? 179 00:13:22,083 --> 00:13:26,363 You remember that snap confines position some environment variables, 180 00:13:26,703 --> 00:13:29,623 right? And there's one which is the snap environment variable. 181 00:13:29,863 --> 00:13:37,043 It turns out that if it's positioned, Kubewe applications are entitled to do stuff behind your back. 182 00:13:38,083 --> 00:13:41,523 One of those things is to say, well, you seem to be running confines, 183 00:13:42,163 --> 00:13:49,403 so I will force a particular platform theme which will talk to the desktop portal. 184 00:13:50,323 --> 00:13:54,963 So, we end up with a very nice dependency at runtime, which is Queen Wayland 185 00:13:54,963 --> 00:13:59,083 loads that platform theme, which tries to talk to XDG Desktop Portal, 186 00:13:59,303 --> 00:14:00,983 which tries to talk to Queen Wayland. 187 00:14:01,963 --> 00:14:05,463 Similarly, we have the XDG Desktop Portal KDE implementation, 188 00:14:05,803 --> 00:14:10,723 which loads the portal platform theme from Qt and tries to talk to itself. self. 189 00:14:12,193 --> 00:14:18,613 Can go well right um this variable was in fact inherited by start plasma you 190 00:14:18,613 --> 00:14:22,813 remember i said that soft plasma pushes some environment the implementation 191 00:14:22,813 --> 00:14:29,993 we have by default is basically it does something like that's all of my environment i put everything there. 192 00:14:31,873 --> 00:14:37,553 Which didn't quite work as we wanted in that in that particular case um so it 193 00:14:37,553 --> 00:14:39,913 does that through the update activation environment code. 194 00:14:40,173 --> 00:14:42,773 So there are several ways to reduce the issue. 195 00:14:43,053 --> 00:14:48,353 So one is to say, well, I control what's in the base map, right, 196 00:14:48,433 --> 00:14:52,873 so I could just shoot the platform theme plugin and then Qt is not able to load it anymore. 197 00:14:53,173 --> 00:14:57,393 That kind of work, but that's not perfect because in fact, Qt reapplication 198 00:14:57,393 --> 00:15:00,573 change its behavior in some other ways, which were not ideal. 199 00:15:01,753 --> 00:15:05,933 So what we are running with right now, the second solution where we have a one-shot 200 00:15:05,933 --> 00:15:10,473 system disservice this, which basically comes just before Queen Wayland, 201 00:15:10,573 --> 00:15:13,433 clean up the environment, and then let everything proceed. 202 00:15:14,373 --> 00:15:19,173 The real fix, which got merged a couple of weeks ago, but we're waiting on the 203 00:15:19,173 --> 00:15:23,413 packaging to catch up, is to modify start plasma to actually be a bit smarter 204 00:15:23,413 --> 00:15:27,553 and say, oh, that's actually stuff for the confinement that doesn't make sense for me to push it up. 205 00:15:29,413 --> 00:15:32,833 So we finally got something working, but that's not confined yet. 206 00:15:34,173 --> 00:15:38,513 So now we want to confine it, and then things break badly again. 207 00:15:39,033 --> 00:15:43,013 And that's mainly because we are talking to systemd directly. 208 00:15:43,473 --> 00:15:47,873 And by default, you're not supposed to talk to systemd because snapd to do its 209 00:15:47,873 --> 00:15:51,693 job actually use some of the systemd facilities and so on. 210 00:15:51,833 --> 00:15:55,353 So obviously, it tries to be between you and systemd and doesn't really want 211 00:15:55,353 --> 00:15:56,313 you to talk to it directly. 212 00:15:57,453 --> 00:16:01,173 In particular, there is no interface in 213 00:16:01,173 --> 00:16:04,093 stock snapd which allows you to use start unit 214 00:16:04,093 --> 00:16:07,873 or update activation environment which are mainly the two things that start 215 00:16:07,873 --> 00:16:13,713 plasma would be calling so when you're like this while you that security right 216 00:16:13,713 --> 00:16:17,153 you take the power drill and then you start making holes so that it starts working 217 00:16:17,153 --> 00:16:22,193 that's pretty much what this is about so we submitted a new interface which 218 00:16:22,193 --> 00:16:23,873 we call systemd user control, 219 00:16:24,393 --> 00:16:30,973 and this one allows weaver as that interface as a plug to be able to call start 220 00:16:30,973 --> 00:16:33,193 unit and update activation environment. 221 00:16:34,693 --> 00:16:38,133 Clearly, this is risky, right? Because then you start talking to systemd directly. 222 00:16:38,313 --> 00:16:40,613 As I said, that's not necessarily something you want to do. 223 00:16:41,893 --> 00:16:45,613 That's a good way for you to manage to actually start something unconfined. 224 00:16:46,533 --> 00:16:50,693 But snap packages using this are actually like very... 225 00:16:51,253 --> 00:16:54,153 There's very few of them, right? There would be the GNOME session because at 226 00:16:54,153 --> 00:17:01,213 some point, they will have the same problems of using the systemd, driving with systemd. 227 00:17:01,773 --> 00:17:07,413 Um, there's, so they just need to be scrutinized, say, right? 228 00:17:10,159 --> 00:17:13,019 But the patch is still in discussion exactly for this, right? 229 00:17:13,099 --> 00:17:17,939 Because it has security implications, so it's going through a very lengthy process 230 00:17:17,939 --> 00:17:20,439 of actually being merged due to this. 231 00:17:21,099 --> 00:17:24,959 Now, there's a note about KIO is that we figured that during the startup, 232 00:17:25,239 --> 00:17:31,839 we also hit KIO KProcessRunner, and that one has the good idea of not calling 233 00:17:31,839 --> 00:17:34,439 start unit, but to call startTransientUnit, 234 00:17:34,619 --> 00:17:38,539 which is a good way to basically execute anything you want, even if that doesn't 235 00:17:38,539 --> 00:17:41,639 have a pre-created service file on the disk. 236 00:17:42,539 --> 00:17:47,999 So, security-wise, this is even more frowned upon than StartUnit itself, 237 00:17:48,319 --> 00:17:54,899 because that opens the door to randomly executing whatever code that weaver wants to execute there. 238 00:17:58,359 --> 00:18:05,779 Luckily, yes, I know, I know, luckily there are still the KDE applications as 239 00:18:05,779 --> 00:18:07,339 forking environment variable. 240 00:18:07,599 --> 00:18:11,659 That does the trick for now. That's probably not what I would recommend to the 241 00:18:11,659 --> 00:18:16,459 community long term, but short to mid-term, I think that's what we want to maintain. 242 00:18:17,279 --> 00:18:22,199 And this way, this forces KIO to go through the old fork-based code path, 243 00:18:22,379 --> 00:18:24,019 and that does the trick currently. 244 00:18:25,479 --> 00:18:28,119 All right. So with that, we can start to confine the session. 245 00:18:29,139 --> 00:18:32,559 So we put a new plug-in in place. 246 00:18:32,719 --> 00:18:37,159 Not everything was working properly. early, we had some issues with case splash and so on. 247 00:18:37,959 --> 00:18:40,759 Long story short, we basically adjusted some more interfaces, 248 00:18:41,159 --> 00:18:46,839 uh, and that's to be expected, right, because they did most of the work on GNOME, 249 00:18:46,939 --> 00:18:51,799 we're using similar interfaces, but with slightly different way of doing it. 250 00:18:51,999 --> 00:18:57,019 Uh, so we find issues, things that they didn't think about when they were putting 251 00:18:57,019 --> 00:18:57,899 in place the interfaces. 252 00:18:59,259 --> 00:19:03,099 Uh, also we are QDBus, we tend to introspect a bit more aggressively and And 253 00:19:03,099 --> 00:19:06,799 generally the introspection is not in the profiles, but that's things which 254 00:19:06,799 --> 00:19:09,719 are trivial to add and they're not too scared about that. 255 00:19:12,879 --> 00:19:16,959 So we improved a bunch of interfaces due to this. 256 00:19:17,739 --> 00:19:22,379 We also declared all the debug services because you have to do that with Snap. 257 00:19:22,999 --> 00:19:27,179 All the names for the services that you will bind to, you have to declare them 258 00:19:27,179 --> 00:19:29,239 all in your Snapcraft. So you did all of that. 259 00:19:29,639 --> 00:19:34,559 And with that, we finally got something which was working from startup to shutdown. 260 00:19:38,039 --> 00:19:41,259 We're happy. Everything goes well. We have all the tasks, right? 261 00:19:41,259 --> 00:19:45,699 Because there's more to that project um and we work on that but something is 262 00:19:45,699 --> 00:19:51,639 not feeling quite right uh and with the comments we get on some of the adjustments 263 00:19:51,639 --> 00:19:55,219 on the interfaces at some point we figure that there's yeah something which 264 00:19:55,219 --> 00:19:59,059 is really not good uh so that's our wake up call so we, 265 00:19:59,719 --> 00:20:05,679 get back to it we use systemd cgls to actually check how the things are confined 266 00:20:05,679 --> 00:20:07,759 in which slice price there are and so on. 267 00:20:07,859 --> 00:20:12,299 And then we figured out, yeah, ah, yeah, we nicely confined Start Plasma. 268 00:20:13,539 --> 00:20:18,119 But then Start Plasma actually started everything as regular services, right? 269 00:20:18,259 --> 00:20:22,199 That's exactly the trap, which I mentioned earlier, right? 270 00:20:23,319 --> 00:20:26,959 So most of the stuff was bypassing Snapd still. 271 00:20:27,179 --> 00:20:31,699 So we went back to the drawing board, added a bunch of aliases on the services 272 00:20:31,699 --> 00:20:35,179 so that we don't provide the regular service files from KDE, 273 00:20:35,279 --> 00:20:38,859 but the ones generated by Snapd because of the daemons we declared. 274 00:20:39,519 --> 00:20:43,619 And then at that point, it actually all goes through Snap. 275 00:20:44,719 --> 00:20:49,159 So we get everything confined, a bit more adjustments we find at that point. 276 00:20:49,499 --> 00:20:53,339 And finally, back on track, we have something which is fully confined. 277 00:20:53,919 --> 00:20:57,659 We have all the services which are confined, including Queen, 278 00:20:57,839 --> 00:21:03,039 Plasma Shell, KDD6, and they all have their own rules for what they can and 279 00:21:03,039 --> 00:21:05,079 cannot do, and that can be completely differentiated. 280 00:21:08,431 --> 00:21:12,071 Now, if you get there, you have PlasmaShell or KRunner and you want to start 281 00:21:12,071 --> 00:21:16,371 apps, right? But the whole point of confining everything is that you cannot call other apps, right? 282 00:21:17,151 --> 00:21:21,111 So if you cannot start apps from PlasmaShell or KRunner, that's kind of pointless. 283 00:21:21,691 --> 00:21:24,471 So how do we even start the other apps from there? 284 00:21:26,411 --> 00:21:31,051 That's actually a snapd feature which is pending. That the people who did the 285 00:21:31,051 --> 00:21:33,191 GNOME side of things, they depend on it as well. 286 00:21:33,271 --> 00:21:40,391 So we just reused that. where Snapd generates an extra desktop file for the 287 00:21:40,391 --> 00:21:43,191 application, which is declared. 288 00:21:43,551 --> 00:21:49,071 In that case, that one which is generated as a specific exec line, 289 00:21:49,291 --> 00:21:52,951 so that's not your regular exec line, which goes through something that they 290 00:21:52,951 --> 00:21:54,631 call snap routine desktop launch, 291 00:21:54,871 --> 00:22:00,171 which we then call into a Dbus interface, 292 00:22:01,571 --> 00:22:07,571 which is exposed by Snapd, and then Snapd will start the actual application for you. 293 00:22:07,651 --> 00:22:11,891 You have access to that Dbus interface only if you have the desktop launch plug, 294 00:22:12,031 --> 00:22:16,551 which we declared just for Plasma Shell and KRunner. 295 00:22:17,591 --> 00:22:21,991 Fun fact, when you start using that one, it horribly breaks if you have the 296 00:22:21,991 --> 00:22:23,711 snap variable, which is positioned. 297 00:22:25,391 --> 00:22:31,031 So if you have that and you try to boot the session and you don't do the dance 298 00:22:31,031 --> 00:22:33,791 of cleanup I mentioned earlier, It doesn't boot, you get the black screen, 299 00:22:33,991 --> 00:22:36,591 but you have to like your black screens at that point. 300 00:22:39,071 --> 00:22:42,671 So that's for the session. We've also been working on the actual, 301 00:22:42,791 --> 00:22:44,371 the few actual applications. 302 00:22:45,091 --> 00:22:47,491 So for the applications, it looks like this. 303 00:22:48,531 --> 00:22:56,871 We have, so two SDKs, one for Qt, one for Qt plus frameworks basically. 304 00:22:57,271 --> 00:23:04,171 And then from this one, we derive the KF6 Core 22, which is a content snap for the runtime, okay? 305 00:23:04,291 --> 00:23:07,931 So at build time, if you're making your own application snap, 306 00:23:08,131 --> 00:23:10,851 at build time, you declare that you need the SDK, 307 00:23:11,611 --> 00:23:15,871 and then at runtime, that you need the content snap for the runtime, 308 00:23:16,151 --> 00:23:20,751 which is exactly what the Neon extension does. 309 00:23:21,611 --> 00:23:26,791 I won't get into the details, but obviously, theming is like a bit of trouble. 310 00:23:27,991 --> 00:23:34,211 They have some systems in Snapd to try to offset that a bit. 311 00:23:34,671 --> 00:23:38,251 But the fact that we provide Q-star as binaries, for instance, 312 00:23:38,451 --> 00:23:41,811 is kind of problematic. That's something which will need to be revisited at some point. 313 00:23:43,045 --> 00:23:46,845 For the applications, as part of the project, we're covering only a few that 314 00:23:46,845 --> 00:23:50,205 you have like the main use cases you might need for an application. 315 00:23:50,425 --> 00:23:54,165 Like seeing images, documents, editing something, saving. 316 00:23:54,885 --> 00:24:00,445 And we also have some sound videos and discover to actually, 317 00:24:00,685 --> 00:24:09,665 because it's a bit particular there, because it needs to, it interacts with Snapd itself in a way. 318 00:24:10,425 --> 00:24:12,505 So we covered all of that. Those 319 00:24:12,505 --> 00:24:17,345 become then good blueprints for other applications to be snapped, right? 320 00:24:17,505 --> 00:24:23,325 That gives enough cases to inspire other people to snap more applications. 321 00:24:23,905 --> 00:24:27,845 The good news is that in our experience, most of them worked out of the box 322 00:24:27,845 --> 00:24:29,945 without any particular work. 323 00:24:30,065 --> 00:24:35,605 Once you declare exactly what it does right inside of the YAML, then pretty much work. 324 00:24:35,985 --> 00:24:38,145 The only exception to that has been Discovered. 325 00:24:38,705 --> 00:24:42,765 We had to patch it a bit because obviously it has a strong proximity with Snapd. 326 00:24:43,505 --> 00:24:48,905 So more things could go wrong. For instance, we were assuming that the packages 327 00:24:48,905 --> 00:24:54,145 were auto-updated all the time, which can be disabled, so we modified that so 328 00:24:54,145 --> 00:24:55,945 that the assumption was not there anymore. 329 00:24:56,605 --> 00:25:00,725 And it was also going through SnapRun to launch the applications which were 330 00:25:00,725 --> 00:25:04,185 installed, which as soon as you put Discover itself in a Snap, you cannot do anymore. 331 00:25:06,757 --> 00:25:12,497 For the apps, we want to have automated tests because we want the snaps to have 332 00:25:12,497 --> 00:25:18,057 functional tests run on them before we even push to the Snap Store. 333 00:25:19,397 --> 00:25:22,397 And so we are trying to have Appium tests for that. 334 00:25:22,917 --> 00:25:26,837 So far, not everything that we have snapped is actually covered. 335 00:25:27,077 --> 00:25:28,957 We are covering Cave Right, Ocular, and Gwenview. 336 00:25:29,577 --> 00:25:32,817 This uncovered a couple of issues with the AT Spy driver. driver, 337 00:25:33,037 --> 00:25:40,337 and so we submitted a couple of simple patches, so problem of interface in the API. 338 00:25:40,777 --> 00:25:46,377 And also, the API tests are you declare the application you want to start, 339 00:25:46,577 --> 00:25:52,197 but we don't want to change the code of the test suite just because it's going 340 00:25:52,197 --> 00:25:55,837 through a snap instead of picking the one which is on your system. 341 00:25:55,977 --> 00:25:58,597 So we needed to adjust to make that a bit easier. 342 00:25:59,737 --> 00:26:03,897 One thing which is blocking to get more of those that we are still seeing some 343 00:26:03,897 --> 00:26:09,197 flakiness from those tests on the CI and that still needs to be investigated. 344 00:26:10,417 --> 00:26:15,557 Talking about CI, we are doing everything to build and publish the snaps. 345 00:26:15,697 --> 00:26:20,517 Obviously, naively, we expected that it would be there when we started the project. 346 00:26:21,137 --> 00:26:27,217 Turned out that someone, I won't name, pulled the plugs out of what was existing 347 00:26:27,217 --> 00:26:30,377 before that. So we had to build something new. 348 00:26:31,377 --> 00:26:35,417 Snapcraft proved to be difficult to run inside of rootless containers, 349 00:26:35,637 --> 00:26:38,177 which is exactly what we use on our CI, of course. 350 00:26:39,357 --> 00:26:45,997 So we had to move everything from now on a VM, which is provided by the KDC's 351 00:26:45,997 --> 00:26:47,357 admins, and thanks for that. 352 00:26:48,317 --> 00:26:52,477 We have the SSH access here. The idea is that it should turn into, 353 00:26:52,577 --> 00:26:56,637 hopefully that will turn into a good blueprint, which can be reused after that 354 00:26:56,637 --> 00:27:00,437 for the sysadmins for ephemeral DMs, at least for use case. 355 00:27:01,833 --> 00:27:05,753 We managed to properly build the snaps, push them all the way to the store. 356 00:27:06,413 --> 00:27:09,953 For now, we went simple, just pushes everything to latest Edge, 357 00:27:10,093 --> 00:27:12,133 because you can have several channels for all the applications. 358 00:27:12,413 --> 00:27:14,453 We plan to support that a bit later. 359 00:27:15,613 --> 00:27:19,593 And once the fakeness issues with the AC SPY web driver are solved, 360 00:27:19,733 --> 00:27:22,713 we will actually block it that the tests don't pass. 361 00:27:24,053 --> 00:27:28,473 That's not all we need to build, right? Because the snaps, so the building blocks, 362 00:27:28,573 --> 00:27:31,493 but we want to build the overall system. So we need to build images. 363 00:27:32,193 --> 00:27:36,233 For that, we use Ubuntu Image, which makes you a system image. 364 00:27:36,893 --> 00:27:43,533 And that was, say, not easy to set up on the CI. The input files that they use need to be signed. 365 00:27:44,273 --> 00:27:47,933 And so we try to have the developers managing their keys locally and so on. 366 00:27:47,953 --> 00:27:49,713 But that's just annoying for everyone. 367 00:27:50,493 --> 00:27:54,553 And then we found out that later on we would need to do things where it actually 368 00:27:54,553 --> 00:28:00,973 needs to be signed with the actual KDE keys. is, so we had to walk away from this. 369 00:28:03,153 --> 00:28:11,693 And so we couldn't rely on Ben as a service to sign the models for us every time we changed them. 370 00:28:11,853 --> 00:28:15,633 So what we have now that we have a manual CI job where we say, 371 00:28:15,793 --> 00:28:20,153 okay, I'm changing the model files, please sign and generate for me, 372 00:28:20,233 --> 00:28:22,953 I still have to download them and so on, but we are fine. 373 00:28:24,113 --> 00:28:26,113 This makes it much easier. 374 00:28:27,373 --> 00:28:30,713 And then we also have to build ISOs, right, because images, full images, 375 00:28:30,853 --> 00:28:33,173 that's nice if you want to pawn with QEMU. 376 00:28:34,113 --> 00:28:37,633 But then if you want to install into like something which is a real machine 377 00:28:37,633 --> 00:28:40,253 or so, you prefer, that's better to have an ISO. 378 00:28:44,387 --> 00:28:50,107 So for that, again, rootless containers, they don't really like this. 379 00:28:50,207 --> 00:28:54,607 Because you need to do, there are a lot of steps involved where you need to 380 00:28:54,607 --> 00:28:56,767 do some root-level things. 381 00:28:57,827 --> 00:29:02,127 At least as we needed to move quickly, especially with the tools we're using, 382 00:29:02,207 --> 00:29:04,507 we need to do bind mounts. And that was a problem. 383 00:29:06,007 --> 00:29:11,907 And so obviously we cannot wait to have the ephemeral VMs on the CI. 384 00:29:13,507 --> 00:29:18,787 All right, so for the challenges, as you can see, we encountered several problems on that project. 385 00:29:19,407 --> 00:29:24,687 So, black screens of various provenance, with and without the confinements, 386 00:29:24,787 --> 00:29:26,087 applications which don't start, 387 00:29:26,267 --> 00:29:30,127 I glanced a bit over that, but that happened, applications which are not doing 388 00:29:30,127 --> 00:29:34,307 what they should, and by not doing what they should, that goes from glitches 389 00:29:34,307 --> 00:29:36,927 to weird behaviors and so on. So, how did we approach those? 390 00:29:37,507 --> 00:29:43,227 If you've got a problem with a Snap application, it's probably worth checking 391 00:29:43,227 --> 00:29:47,407 first if, well, can you get that outside of Ubuntu Core as well, right? 392 00:29:47,467 --> 00:29:49,947 When you use the Snap, like, on your host. 393 00:29:51,447 --> 00:29:56,447 And if you still see the problem, then that's probably something with your application and the Snap. 394 00:29:57,187 --> 00:30:00,947 So you have to troubleshoot. You can use some tooling there. 395 00:30:01,047 --> 00:30:05,687 So Snappy Debug helps quite a bit. It even suggests fixes when you have problems 396 00:30:05,687 --> 00:30:07,547 with the SAP component app armor. 397 00:30:08,187 --> 00:30:12,787 Then you can introspect with Snap-run Shell the actual environment which is 398 00:30:12,787 --> 00:30:14,327 seen without starting the app. 399 00:30:14,427 --> 00:30:18,907 So, you can be inside of the confinement and see how your application will see the system. 400 00:30:19,127 --> 00:30:21,147 You can trace into something. 401 00:30:21,507 --> 00:30:26,167 You can have GDB server and then, well, you need debug infoD, clearly. 402 00:30:28,462 --> 00:30:31,622 So that's nice. I mean, there's some tooling, right? Now, if you have a problem 403 00:30:31,622 --> 00:30:34,242 with the session, well, you can still use snappy debug. 404 00:30:35,062 --> 00:30:39,882 The only problem is that the proposed fixes are generally a lie. 405 00:30:41,922 --> 00:30:45,222 And that makes sense, right? It's been written from the perspective of making 406 00:30:45,222 --> 00:30:46,702 an application, not a session. 407 00:30:46,862 --> 00:30:51,222 So very often it gives you the opposite solution that you would need, 408 00:30:51,302 --> 00:30:54,342 right? Because you're not sitting on the same side of the fence. 409 00:30:55,322 --> 00:30:57,002 Apart from that, you're a bit on your own. 410 00:30:58,502 --> 00:31:02,822 There won't be any specific tooling which will help, so you have to go lower levels. 411 00:31:03,762 --> 00:31:09,462 You can roll your own Plasma Desktop session snap, the one I mentioned earlier, 412 00:31:09,702 --> 00:31:13,202 that's just a bunch of scripts, so you could always add some more environment 413 00:31:13,202 --> 00:31:15,722 variables, clear things inside of the script, 414 00:31:16,022 --> 00:31:19,522 install that, reboot the session, and then you get more logs, basically. 415 00:31:21,602 --> 00:31:27,022 You can also roll out your own Plasma Core Desktop snap if things go really, 416 00:31:27,142 --> 00:31:32,022 really bad, and you cannot even log in the system or get through any logs or anything like this. 417 00:31:32,262 --> 00:31:36,602 So we're providing a small script to make that easier, which enables a bunch 418 00:31:36,602 --> 00:31:42,962 of things like, for instance, having a passwordless root access on the serial port. 419 00:31:43,602 --> 00:31:46,282 That's clearly just for the developers locally to use. 420 00:31:48,062 --> 00:31:52,122 All of this is immutable and you need to keep that in mind at every step. 421 00:31:52,382 --> 00:31:55,702 That means you have to be strategic in what you prioritize when you introspect, 422 00:31:55,702 --> 00:31:59,602 because there are things you can iterate quickly on, like the applications, 423 00:31:59,762 --> 00:32:04,162 like the session snap, or even snapd itself that you can actually reinstall. 424 00:32:05,602 --> 00:32:10,142 At best, you restart the session with the session snap. 425 00:32:11,282 --> 00:32:15,402 From time to time, you might need a reboot with snapd, but not very often in my experience. 426 00:32:15,982 --> 00:32:20,182 But if you need to touch the base snap, then it takes a while, right? 427 00:32:20,782 --> 00:32:22,962 And you can go get a coffee and everything. 428 00:32:23,902 --> 00:32:27,382 This is a much slower feedback loop, so it's better to not do that often. 429 00:32:28,552 --> 00:32:33,052 There are current limitations with what we are doing, mainly because of the 430 00:32:33,052 --> 00:32:35,972 unmerged interface, which is still in discussion. 431 00:32:36,372 --> 00:32:39,552 Because it's unmerged, so the official Snapd doesn't have it. 432 00:32:39,612 --> 00:32:42,952 So we have to ship like Snapd, which has that on top. 433 00:32:44,372 --> 00:32:50,492 That's fine. We managed to get that in the store. We are collaborating with Canonical for that. 434 00:32:50,992 --> 00:32:54,632 That also means that you cannot have an accession on the Snap store, 435 00:32:54,792 --> 00:32:57,172 which says, yeah, I allow you to have that interface. interface, 436 00:32:57,292 --> 00:33:01,532 so you cannot really publish on the store, which makes it very interesting for 437 00:33:01,532 --> 00:33:03,552 people to actually get it when they build their image. 438 00:33:04,592 --> 00:33:08,512 So even on the CI right now, we're injecting like a local build of what we did 439 00:33:08,512 --> 00:33:11,932 straight in the image, and that's far from ideal. 440 00:33:12,952 --> 00:33:17,412 And that's not the end of it, because then at runtime, nothing will auto-connect 441 00:33:17,412 --> 00:33:20,252 to Plasma Desktop session because of that. 442 00:33:21,072 --> 00:33:25,972 So you need to connect manually. This isn't great. We hope that it's getting 443 00:33:25,972 --> 00:33:29,912 slowly resolved, and we hope that soon we will have a nicer experience due to this. 444 00:33:30,252 --> 00:33:34,052 There's some upcoming work. So one is currently, as you've seen, 445 00:33:34,232 --> 00:33:37,892 that was Core 22 everywhere. We have to do the switch to Core 24. 446 00:33:38,952 --> 00:33:45,752 And so we were a bit waiting for a snapshot of Numbat for KDE Neon for that. 447 00:33:46,472 --> 00:33:50,132 So we have this, so we should be able to start soon. 448 00:33:51,752 --> 00:33:55,572 That shouldn't impact our architecture much, but then that means that plenty 449 00:33:55,572 --> 00:34:00,952 of stuff would change at the same time, and we expect a lot could go wrong. 450 00:34:02,712 --> 00:34:06,832 By doing this, we will also have a better approach for the provisioning because 451 00:34:06,832 --> 00:34:10,212 on Core 24, they provide some services for that, so hopefully we will do better 452 00:34:10,212 --> 00:34:11,172 than what we are doing now. 453 00:34:11,172 --> 00:34:17,112 Now, the big one is that even the GNOME side are moving to a more modular architecture 454 00:34:17,112 --> 00:34:20,572 because the fact that we are stuffing everything in the base map, 455 00:34:20,712 --> 00:34:22,772 that's kind of a problem for composability, right? 456 00:34:22,912 --> 00:34:27,152 That's kind of a problem for reuse of what the fixes that they do on Core 24. 457 00:34:28,152 --> 00:34:31,792 So there will be an effort to actually split things up a bit further. 458 00:34:32,792 --> 00:34:37,512 Not everything is set in stone, but the idea is that we will probably have like 459 00:34:37,512 --> 00:34:39,932 a content snap, so everything will be in different prefix. 460 00:34:41,172 --> 00:34:46,652 And then the session snap and this ddm referring referring to this where we 461 00:34:46,652 --> 00:34:50,632 are not sure yet is that do we want the plasma core content snap to actually 462 00:34:50,632 --> 00:34:55,852 reuse the runtime bits of kd frameworks or do we want to separate one that's 463 00:34:55,852 --> 00:34:57,272 something we still have to figure out, 464 00:34:59,034 --> 00:35:04,534 a few lessons learned. I never really liked packaging, I confess. 465 00:35:05,534 --> 00:35:09,974 But I have to admit that for Snap, the Snapcraft rules are not too bad. 466 00:35:10,194 --> 00:35:14,654 The extension that we introduced actually helps quite a bit. 467 00:35:15,014 --> 00:35:16,354 That wasn't too horrible. 468 00:35:19,174 --> 00:35:23,734 The Ubuntu Core desktop architecture is still a bit in a state of flux, right? 469 00:35:25,334 --> 00:35:29,094 So, that means things are less documented. implemented, the tools of shopper 470 00:35:29,094 --> 00:35:33,794 ages, and where we basically jump ourselves on every knife we find. 471 00:35:35,174 --> 00:35:39,454 And the behavior enforced by the SnapStore that actually makes things slower 472 00:35:39,454 --> 00:35:45,394 for the development, for instance, the interface which is on hold currently. 473 00:35:46,814 --> 00:35:51,334 We have good tooling nowadays to debug systemd issues, and we are happy about that. 474 00:35:51,954 --> 00:35:56,894 You have to double check, triple check when you confine. But if you screw it 475 00:35:56,894 --> 00:35:59,574 up and you didn't confine, that's fine because really you don't want to confine 476 00:35:59,574 --> 00:36:04,134 much and then you want to confine more over time, which in the end is what we did. 477 00:36:04,734 --> 00:36:09,874 And we should definitely start to think about ways of avoiding strat-transient 478 00:36:09,874 --> 00:36:15,714 units or provide fork-based alternative or service like GVFS or something like this. 479 00:36:17,914 --> 00:36:22,594 We can't afford to completely deprecate and not provide anything if we want 480 00:36:22,594 --> 00:36:24,194 to be serious about the sandboxing. 481 00:36:25,114 --> 00:36:30,934 To contribute, that's all on Invent. So there's the Ubuntu core package project, 482 00:36:31,214 --> 00:36:32,354 the Snap Packaging Group. 483 00:36:32,554 --> 00:36:38,174 There's plenty there. We have some guidelines, how-tos, and there's a KDN on 484 00:36:38,174 --> 00:36:40,454 core page. So you can come talk to me. 485 00:36:42,014 --> 00:36:46,974 I'm just taking the time left to thank the team at Ineokia Haute Couture, 486 00:36:47,034 --> 00:36:51,994 which helped with that, and also the contractors we've been working with. 487 00:36:51,994 --> 00:36:55,354 So Scarlett Moore is unfortunately not here this year. 488 00:36:55,474 --> 00:36:59,034 And Carlos de Mane is probably somewhere around here. 489 00:36:59,194 --> 00:37:03,694 And I'd like also to thank some of the people who provided help on their spare 490 00:37:03,694 --> 00:37:09,334 time and bounced ideas, like Harald and David Edmondson, or Ben, 491 00:37:09,474 --> 00:37:11,374 which I annoyed a lot with the CI. 492 00:37:13,634 --> 00:37:16,654 Thanks for your attention. I'm not sure we have time for questions, 493 00:37:16,814 --> 00:37:21,054 but... You have 60 seconds for questions. Okay. That's called one question then. 494 00:37:24,334 --> 00:37:26,454 All right thank you.