[Cloud-packages] Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper

Sat Aug 20 20:12:25 UTC 2016

I forgot..
You also need to apply *, if you want to use --hdd-size auto with ext{3,4}, and ** for syslinux to work.

I use HDD images, so if something break in newer Debian version (currently we use Jessie), I most likely gonna fix it.

* https://anonscm.debian.org/cgit/debian-live/live-build.git/commit/?id=25b15bce09e4d64e4f7e29c573308a8dd0135a05
** https://anonscm.debian.org/cgit/debian-live/live-build.git/commit/?id=a8cc7084279116870bc0e37e0836ac7e1f5b780a

Regards Kristian

----------------------------------------
> From: klausenbusk at hotmail.com
> To: hartmans at debian.org; debian-cloud at lists.debian.org; debian-cd at lists.debian.org
> CC: vmdebootstrap at packages.debian.org; bootstrap-vz at packages.debian.org; debian-live at lists.debian.org; klee at debian.org
> Subject: RE: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
> Date: Sat, 20 Aug 2016 22:07:44 +0200
>
> Hello Hartman
>
>> You can do some things by selecting plain as the root filesystem, and
>> hdd as the image type, but I seem to recall that doesn't end up
>> working.
>
> We have used HDD images for around 1 year without any (big) problems,
> if you use a newer version of e2fsprogs (1.43<), you need to apply this * patch
> which I pushed  ~ 2 months ago.
>
>
> * https://anonscm.debian.org/cgit/debian-live/live-build.git/commit/?id=489a09ba92328cef846598a15329e9ff91519e3c
>
> Regards Kristian Klausen
>
> ----------------------------------------
>> From: hartmans at debian.org
>> To: debian-cloud at lists.debian.org; debian-cd at lists.debian.org
>> CC: vmdebootstrap at packages.debian.org; bootstrap-vz at packages.debian.org; debian-live at lists.debian.org; klee at debian.org
>> Subject: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
>> Date: Tue, 16 Aug 2016 11:56:12 -0400
>>
>>
>> Please note the follow-up to the cloud list.
>>
>> Hi.
>>
>> I recently finished putting together a custom image of what amounts to
>> stretch for work. Today, this is mostly not a cloud image, although
>> that's expected to change for our future customer deployments.
>>
>> For my previous job I did maintain a cloud image using live-build
>> through the wheezy and jessie cycles.
>>
>> In this discussion I'm focusing on the needs of custom images. We've
>> all agreed that a robust and flexible approach for custom images is an
>> important goal of our cloud (and presumably live) efforts.
>> My personal belief is that we should use the same tools for official
>> images as we recommend for custom images. Back in the days when I cared
>> about sets of CDs I always found it frustrating that the debian-cd
>> packages could make official images, but you were better off with other
>> approaches for custom cd sets. There would be unfortunate and undesired
>> differences introduced because you weren't using debian-cd, but it was
>> never worth the pain of hacking debian-cd for custom sets.
>>
>> As a user, I'd like to contrast this significantly with live images,
>> where we have (so far) used mostly the same tooling for official and
>> unofficial images.
>> It feels more polished and the approach is better.
>>
>> I'd like to get a couple of things out of the way first.
>> live-wrapper
>> I think live-wrapper is too immature to evaluate. I tried to evaluate
>> it as a replacement for live-build, but quickly concluded that it's too
>> early in the live-wrapper development cycle for me to be able to look
>> at. The documentation of live-wrapper 0.3, which I looked at, was very
>> sparse. It requires a lot of the directory structure where you run it
>> without clearly documenting that. There's some sort of customization
>> script that is needed, but it's unclear how that works.
>>
>> Live-wrapper is missing functionality that was critical to me in my use
>> of live-build. It does not support live-installer. It does not support
>> including d-i and a live system on the same CD. It didn't appear to
>> support easy customization of the bootloader configuration.
>>
>>
>> I think it's great that people are looking at something built on top of
>> vmdebootstrap. I hope that effort continues, and when it's more mature
>> I'd be happy to take another look.
>>
>> I'd advise taking a careful look at my comments on vmdebootstrap, as I
>> think several of them may have implications for live-wrapper.
>>
>> bootstrap-vz
>>
>> Bootstrap-vz is a collection of python for building cloud and VM
>> images. I read all the documentation, examined some chunks of the code,
>> but did not end up using the product to build an image.
>>
>> Up front I'll say I was disappointed that the code is python2.7 not
>> python3. It actually does matter. As I build a wider collection of
>> idiomatic python3 within my own domain, it would be valuable to be able
>> to use that while customizing the images I develop. Making it
>> impossible for me to use my code and the bootstrap-vz libraries and
>> plugins in the same process is architecturally disappointing.
>>
>> The strength of bootstrap-vz is that it has a plugin architecture, and
>> supports tweaking aspects of the image for the environment. There's an
>> assumption that things will be different between ec2, gce, virtualbox,
>> kvm, etc. You have to pick one of these at a fairly basic level, and it
>> significantly affects the code path that will be used.
>>
>> There are a bunch of plugins that accomplish things. The code supports
>> images with and without partition tables. It supports images backed by
>> EBS volumes, raw devices, and even things like vdi for VirtualBox.
>>
>> Different virtualization layers can plug in plugins. For example the
>> kvm layer plugs in support for virtio. This is both a blessing and a
>> curse. Virtio is a great example: VirtualBox supports it too, and it's
>> unclear why that plugin would be introduced at that level in the code.
>>
>> Bootstrap-vz failed to meet my needs in two important ways. First, I
>> found it too difficult to audit how an image I would create with
>> bootstrap-vz would iffer from what I'd get debootstrapping something by
>> hand. That is, I found that I could not trust the changes bootstrap-vz
>> might introduce for me. I'd end up needing to read most of the code to
>> evaluate this, because it seems like a major goal of the design is to
>> make those sorts of tweaks easy.
>>
>> Secondly, with some irony, I realized that it would be annoying for me
>> to add my own customizations to the image. Bootstrap-vz makes it easy
>> for bootstrap-vz developers to add new tweaks, but makes it challenging
>> for custom image builders who don't want to hack on the bootstrap-vz
>> sources. I really want something like the vmdebootstrap customization
>> script where I can run a chunk of my shell or Python code at appropriate
>> points in the process. Secondly, even if I did want to write
>> vmdeboothstrap-style tasks and plugins, it looks like getting them
>> integrated might be a bit tricky. Bootstrap-vz uses a lot of plugins,
>> but the plugin consumers seem to know way too much about the plugins
>> they consume. Take a look at bootstrapvz/providers/kvm/__init__.py to
>> see what I'm talking about. That file knows about the kvm-specific
>> tasks; it integrates them into the flow, knows about their
>> configuration, etc. Instead I'd hoped for an architecture where the
>> consumer might sometimes be responsible for loading the plugin, but the
>> integration of the plugin into the process would be accomplished by the
>> plugin registering itself. That is, in the mentioned __init__.py file,
>> the import statements seem fine, but the parent module knowing so much
>> about the child module seems undesirable in a plugin design.
>>
>> While I did not use Bootstrap-vz, I want to call out a bunch of
>> excellent properties. It does have a plugin architecture. It
>> demonstrates thoughtful enough design that you can integrate new
>> programatic bits into a structure, sharing code, but using a somewhat
>> declarative approach rather than a purely procedural approach. That is,
>> the objects and plugins seem used to relatively good effect. While i
>> complained about lack of extensibility for out-of-tree users,
>> extensibility for in-tree developers seems to be a huge strength.
>>
>> Also, the flexibility is a huge strength. To the extent we need to
>> tweak things based on environment, provider and the like, bootstrap-vz
>> is good.
>>
>> Finally, in terms of the resulting image, the manifest is purely
>> declarative. I chafe at that building custom images, but for official
>> images that's probably a huge strength.
>>
>> Live Build
>>
>> Live Build is the most comprehensive image building tool that I've
>> used. It has very good documentation compared to the other tools. It
>> supports a rich declarative format: I can add additional archives, keys,
>> packages and the like. It also supports hooks so I can make
>> customizations as needed.
>>
>> It has caching infrastructure. I found that performance with live-build
>> was much better than vmdebootstrap.
>>
>> Live Build has critical functionality that is missing from other
>> solutions. I talked about the aspects of that functionality that relate
>> to live systems under live-wrapper, the only other tool I looked at that
>> tries to handle live systems directly. I'll talk about the other
>> missing functionality under vmdebootstrap.
>>
>> Unfortunately, Live Build has some significant disadvantages. It's
>> written in shell. It is a triumph of modular shell programming. I
>> found that given time, I was able to understand an debug it: given how
>> complex the shell scripts are, that says something very positive. I
>> found that even when some of what it wanted to do wasn't right for my
>> needs, code reuse was possible. Between the phase commands, function
>> libraries and similar, lots of thought was put into modularity.
>>
>> However, triumph of modular shell programming though it is, it IS ALL
>> SHELL. There's only so modular a shell script can be. There weren't
>> really objects—and while yes, I know that you can do object-oriented
>> shell, it doesn't make things better.
>>
>> Live Build doesn't do a good job at things that aren't live systems.
>> You can do some things by selecting plain as the root filesystem, and
>> hdd as the image type, but I seem to recall that doesn't end up
>> working. I ended up using only part of the typical chain (lb
>> bootstrap&&lb chroot rather than lb build, then writing my own short
>> script to make the image).
>>
>> Live Build was starting to reach the maximum complexity of its design.
>>
>> Live Build had a lot of case statements scattered throughout the code.
>> If the image type is iso, then we'll go through all the boot loader
>> options. If the ibmage type isn't iso, we'll do this other thing. This
>> meant that adding a new option involved going and making sure you'd
>> added all the case statements to deal with all the possibilities.
>>
>> Live Build didn't have enough of a tesnt suite. It wasn't really
>> reliable. Upgrading was always filled with fear and uncertainty.
>>
>> Live Build is not currently maintained.
>>
>> That said, if I had an existing Live Build project, I would not migrate
>> away from Live Build today. I'd expect to try and keep it limping along
>> through Stretch, fighting anyone who tries to remove Live Build,
>> possibly throwing in some Live Build bug fixing along the way. I'd plan
>> for probable migration in the Buster time frame though.
>>
>> vmdebootstrap
>>
>> vmdebootstrap is relatively limited in what it does, but IT WORKS. The
>> first time I ran it, it produced a working image. That's very unusual
>> in my experience of image creation tools.
>>
>>
>> vmdebootstrap is relatively focused in its image creation. It will loop
>> mount a msdos-partitioned image that it creates. It will run
>> debootstrap with one filesystem and an optional boot partition or UEFI
>> ESP partition. It will install a boot loader.
>>
>> It has a few custom hacks, mostly only for older releases. Fore example
>> you can request serial console, but not for stretch. You can request
>> dhcp, although be warned that with stretch you'll get systemd-networkd,
>> which may surprise you.
>>
>> All configuration is on the command line.
>>
>> You can run one customization script during image creation.
>>
>> There are some huge strengths. Development is active. I think all but
>> one of the bugs I reported was fixed before writing this comparison.
>> It works. There is an active culture of regression testing and
>> continuous integration.
>>
>> It supports UEFI. It only does DOS partition tables not GPT , which is
>> kind of surprising, but support for that is mandated by the UEFI spec.
>> I've had some annoyance with real hardware having to reconfigure the
>> BIOS to prefer UEFI boot because of this decision, but it does work.
>> However, there are some huge defects.
>>
>> Extensibility is lacking.
>> It, like bootstrap-vz, is written in python2. Another problem makes
>> that less of an issue than it is for bootstrap-vz: vmdebootstrap is not
>> particularly extensible procedural code. There are modules, but there's
>> not a good object hierarchy, and there are neither particularly reusable
>> components (outside of some libraries for running commands and the like)
>> nor are there points at which you could integrate plugins. So, I'm
>> much less likely to want to use my own Python libraries in vmdebootstrap
>> than I am with bootstrap-vz.
>>
>>
>> You get that one customization point. If it's at the wrong place for
>> your needs, guess you'll have to deal somewhere else. It turns out that
>> I needed to change the image after update-grub had been run for the last
>> time. I want to make sure that each of my installed machines ends up
>> with a different BTRFS filesystem uuid for the root. If I ever want to
>> take a snapshot of one machine and manipulate it on another, that
>> becomes important. So, I have written an initramfs-hook to change the
>> uuid on the root (you can't do that to a mounted btrfs) and fixes up
>> /etc/fstab. I trigger this by a root command line of the form
>> root=REUUID= instead of root=UUID=. So, I want to substitute UUID= with
>> REUUID= in the grub configuration for the first boot. I definitely do
>> not want that change to survive a run of update-grub (which I'll arrange
>> for on first boot).
>>
>> So, I ended up writing Python code to mount the image, bypassing
>> vmdebootstrap's customization entirely. I con-considered borrowing code
>> From filesystem.py. However, that code depends on the global settings
>> dictionary that is maintained by the command line script. Those bits
>> weren't really reusable. (And also vmdebootstrap seems to leak loop
>> devices; I had to write wrappers to clean that up)
>>
>> It's telling that live-wrapper, even though it is also Python, doesn't
>> use vmdebootstrap as a library. It calls the command line script.
>> There's no other way you could do it: vmdebootstrap's extensibility
>> profile isn't good enough.
>>
>> If you want LVM, too bad. If you want GPT, too bad. If you want your
>> root filesystem labeled, better do that yourself.
>>
>> Squashfs generation is supported, although you are responsible for
>> setting up live-boot and live-config. You may have a bit of fun
>> integrating that with vmdebootstrap's bootloader logic. (You might
>> think that live-wrapper would at least do that for you. Not yet. I'm
>> actually totally fine with vmdebootstrap's decision to push configuring
>> live-* up a layer, although I think it needs better customization hooks
>> to make that work well.)
>>
>> vmdebootstrap doesn't have support for adding keys to Apt's trusted key
>> store. It doesn't have support for adding additional archives to your
>> sources. Debootstrap requires that all the packages come from one
>> mirror. vmdebootstrap doesn't provide a way to install packages from
>> another mirror later. Of course you can work around all that—in the
>> customization hook you're provided even. However, that's all critical
>> functionality you expect from a custom image tool. Some layer needs to
>> provide that for me if it's really going to be a complete solution for
>> managing custom images.
>>
>> SOURCE matters. If I'm building an image that I plan to distribute to
>> people, GPL and other licenses require that I distribute source. “But
>> Sam, you have the Debian mirror you used.” And so I do, until it
>> changes. I tend to distribute images longer than mirrors stay stable.
>> Yes, I could use aptly to snapshot my entire mirror. Yes, I could
>> coordinate s,m.something to figure out what packages make their way into
>> the image and snapshot only those package sources. Live Build gives me
>> this, and at least for custom images, that too is a requirement.
>>
>> Ultimately, I did produce a reliable tool that calls vmdebootstrap under
>> the covers and then does a bunch of
>> customizations. Am I glad I used
>> vmdebootstrap rather than fighting with
>> Live Build? Eh. I learned something,
>> and I'm not really unhappy with my
>> choices. But no, vmdebootstrap is not a
>> clear win today. Tomorrow?? Now, that is
>> the question.
>>
>>
>> A Suggestion
>>
>> As a user, who would love to see Debian be great, but who has no
>> political skin in any previous battles, I'd like to offer a suggestion
>> for something to explore.
>>
>> I'd like to see some folks explore bootstrap-vz layered on top of
>> vmdebootstrap. Debootstrap would be responsible for taking Debian
>> packages and unpacking and doing initial configuration, just as it
>> always does. vmdebootstrap would be responsible for partitioning,
>> filesystem creation, bootloader installation and running debootstrap.
>> Bootstrap-vz would be responsible for tweaking the image, for providing
>> a framework for those tweaks, and for the schema/manifest language to
>> describe what it is that we want in an image.
>>
>> It would require compromises and changes all around.
>>
>> Do EC2 eimages really need to be built in an EBS volume? Would it be
>> good enough to copy them there after, building in a loop file like
>> everything else until that moment?
>>
>> vmdebootstrap would need to gain an API and an object model.
>>
>> We'd probably have to give up some of the tweaks we have, and add
>> support either for plugins for some of the more basic tweaks directly
>> into vmdebootstrap. As an example, vmdebootstrap would almost certainly
>> need to support raw images without a partition table.
>> However, I think an clear interface between vmdebootstrap and
>> bootstrap-vz would make it easier to audit what was being done to an
>> image and might improve both products.
>>
>> I think we should do something. If there's one thing I hope you take
>> away from this message, it's that none of the options I explored is
>> ready—probably not even for Stretch. Will we have a regression in our
>> custom image functionality Stretch over Jessie? Will we have a solution
>> in time for Stretch? I don't know the answers to either of those, but I
>> hope that we can come together in a spirit of cooperation and work on
>> those questions rather than focusing on parallel, independent paths.
>