Sunday Aug 16, 2020 15:37, last edition on Sunday Aug 16, 2020 15:42
For my Linux containers, I am using LXD.
The recommended way to install LXD is Snap, which updates daily.
Few days ago, the last release of LXD has been released and the update started automatically.
For the next two following days, I experienced timeout on my virtual servers.
I also noticed at lot of input/output (I/O) on the hard drives.
I started to investigate.
I identified, using Htop among other tools like Netdata, that the process snapd, from Snap, was using 100% of the hard drives and all the I/O were on the /var/ partition.
Using the command snap watch --last auto-refresh, I saw that the update stated more than two days ago and was stuck at the step Copy snap "lxd" data.
I aborted the ongoing procedure but I ended up with a completely broken LXD.
Hopefully, on Linux Containers forum, I found someone with the same, or at least similar enough, issue.
You can try to automate snap in offline mode. Make sure your host is not connected to the internet. This is definitely recommended for hypervisors and therefore also the LXD hosts. (my opinion for LXD hosts)
Download snap somewhere else with:
snap download lxd
Copy the files to your LXD node and install the snap:
snap ack <package.assert>
snap install <package.snap>
It saved my day! At least, what was left ot it.
It is at least the second time that this issue happens to me and it appears to impact other user.
Unfortunately, it is a sign that LXD is not ready for production environment which requires stability.
It is also a reminder that I need to find a way to backup LXD.
Monday Jul 27, 2020 12:05
Feedback on LXD
I had an issue with LXD where its database got corrupted.
In that case, there is nothing we can do.
All Linux containers on every physical servers are gone.
I managed to restore most of the containers using rsync, but the internal file-system management of LXD is completely screwed.
I cannot export not publish my containers to backup them, nor restart my LXD configuration from scratch by exporting and importing them back.
My only option is to get another physical server, I have three so far, so adding a fourth one, make a new LXD server, outside of the current cluster.
Then, recreate from scratch all my containers, which will be a a task requiring a tremendous amount of time.
After this, deleting the current LXD cluster,
creating a new one with the three physical servers,
export the containers from the temporary LXD server
and finally importing the containers to the cluster.
It is necessary that I perform this task as quickly as possible because I cannot perform backup.
Adding 220.127.116.11 to iptables (INPUT, DROP) has not enough.
I added other IPs and few range of IPs, up to /16.
It was not enough and it was a fastidious task.
So I added a new field to the comment section.
You need to answer yes.
It is not case sensitive.
This simple trick appears to do the job so far.
My git repo were not working anymore.
I forgot to enable the mods in Apache HTTP Server when I migrate from Xen to LXD.
a2enmod cgi alias env
It is now fixed.
For more information, see
Statistics were not working anymore.
I use a custom version of Awstats in order to integrate smoothly the statistics in my website.
Part of this script rely on the package provided by Ubuntu.
When I upgrade the virtual machine from Ubuntu 16.04 LTS (Xenial Xerus) to Ubuntu 18.04 LTS (Bionic Beaver),
some modifications in Awstats broke my customization.
I realize that maintaining this part will be complex as it is a 22000+ lines Perl file which is difficult to edit automatically.
I made a quick fix.
It will most-likely broke again when I will migrate from Ubuntu 18.04 LTS (Bionic Beaver) to Ubuntu 20.04 LTS (Focal Fossa).
To Do List
I am working on TootLine, the PHP code that allows you to share your TootLine on your blog, like the one there is on the right or bottom, depending on the size of your screen. I have couple of issues to address before publishing it, which are proper word wrapping, create a cache for the media in order to solve CSP issue, handle the NSFW content that is displayed so far.
I would like to make a French version of this blog, with most of the articles translated.
More restrictive CSP headers
I want to rewrite some part of the web site to be able to provide more secure CSP headers.
I am planning on adding a RSS feed for each commentary section so it will be easy to follow. I will also add a cookie to auto fill the fields Name and Website. I will put a check box if you want to add the cookie when you comment.
I will add the list of all tags on the right panel.
I will add proper Open Graph protocol and Twitter cards in the headers.
I already updated the MySQL database, so everything is ready on this side,
I just need to rewrite the headers that I include to make them dynamic.
Better looking links
I already changed the URL of some links to make them better looking, but I did not finish yet.
The rewriting rules are not as simple as I expected,
if you want to make them SEO compliant.
My work-flow is not the most efficient.
Each article points to an actual file, which is not so good, because I need to create a file each I add an entry in this blog.
I will improve this soon.
It is one of the reason why I stopped writing here.
It is too complex.
Migrating the last virtual from Xen to LXD
I need to migrate completely my photo galleries (Piwigo) from XEN to the new ones on LXD.
Home made modem/router/NAS
I bought few items in order to build my own modem/router/NAS.
Although RSS feed appears to work fine, there are PHP errors in the logs.
I need to investigate.