Dr Clèm's Blog

How to fix a broken LXD server due to fail snap auto-refresh

Sunday Aug 16, 2020 15:37, last edition on Sunday Aug 16, 2020 15:42

For my Linux containers, I am using LXD. The recommended way to install LXD is Snap, which updates daily. Few days ago, the last release of LXD has been released and the update started automatically. For the next two following days, I experienced timeout on my virtual servers. I also noticed at lot of input/output (I/O) on the hard drives. I started to investigate. I identified, using Htop among other tools like Netdata, that the process snapd, from Snap, was using 100% of the hard drives and all the I/O were on the /var/ partition. Using the command snap watch --last auto-refresh, I saw that the update stated more than two days ago and was stuck at the step Copy snap "lxd" data. I aborted the ongoing procedure but I ended up with a completely broken LXD. Hopefully, on Linux Containers forum, I found someone with the same, or at least similar enough, issue.

You can try to automate snap in offline mode. Make sure your host is not connected to the internet. This is definitely recommended for hypervisors and therefore also the LXD hosts. (my opinion for LXD hosts)

Download snap somewhere else with:
snap download lxd

Copy the files to your LXD node and install the snap:
snap ack <package.assert>
snap install <package.snap>

Linux Containers

It saved my day! At least, what was left ot it. It is at least the second time that this issue happens to me and it appears to impact other user. Unfortunately, it is a sign that LXD is not ready for production environment which requires stability. It is also a reminder that I need to find a way to backup LXD.

Captain's Log #5

Monday Jul 27, 2020 12:05
Feedback on LXD

I had an issue with LXD where its database got corrupted. In that case, there is nothing we can do. All Linux containers on every physical servers are gone. I managed to restore most of the containers using rsync, but the internal file-system management of LXD is completely screwed. I cannot export not publish my containers to backup them, nor restart my LXD configuration from scratch by exporting and importing them back. My only option is to get another physical server, I have three so far, so adding a fourth one, make a new LXD server, outside of the current cluster. Then, recreate from scratch all my containers, which will be a a task requiring a tremendous amount of time. After this, deleting the current LXD cluster, creating a new one with the three physical servers, export the containers from the temporary LXD server and finally importing the containers to the cluster. It is necessary that I perform this task as quickly as possible because I cannot perform backup.

Spam, again

Adding to iptables (INPUT, DROP) has not enough. I added other IPs and few range of IPs, up to /16. It was not enough and it was a fastidious task. So I added a new field to the comment section. You need to answer yes. It is not case sensitive. This simple trick appears to do the job so far.


My git repo were not working anymore. I forgot to enable the mods in Apache HTTP Server when I migrate from Xen to LXD.

a2enmod cgi alias env
It is now fixed. For more information, see Smart HTTP.


Statistics were not working anymore. I use a custom version of Awstats in order to integrate smoothly the statistics in my website. Part of this script rely on the package provided by Ubuntu. When I upgrade the virtual machine from Ubuntu 16.04 LTS (Xenial Xerus) to Ubuntu 18.04 LTS (Bionic Beaver), some modifications in Awstats broke my customization. I realize that maintaining this part will be complex as it is a 22000+ lines Perl file which is difficult to edit automatically. I made a quick fix. It will most-likely broke again when I will migrate from Ubuntu 18.04 LTS (Bionic Beaver) to Ubuntu 20.04 LTS (Focal Fossa).

To Do List

I am working on TootLine, the PHP code that allows you to share your TootLine on your blog, like the one there is on the right or bottom, depending on the size of your screen. I have couple of issues to address before publishing it, which are proper word wrapping, create a cache for the media in order to solve CSP issue, handle the NSFW content that is displayed so far.


I would like to make a French version of this blog, with most of the articles translated.

More restrictive CSP headers

I want to rewrite some part of the web site to be able to provide more secure CSP headers.


I am planning on adding a RSS feed for each commentary section so it will be easy to follow. I will also add a cookie to auto fill the fields Name and Website. I will put a check box if you want to add the cookie when you comment.


I will add the list of all tags on the right panel.


I will add proper Open Graph protocol and Twitter cards in the headers. I already updated the MySQL database, so everything is ready on this side, I just need to rewrite the headers that I include to make them dynamic.

Better looking links

I already changed the URL of some links to make them better looking, but I did not finish yet. The rewriting rules are not as simple as I expected, if you want to make them SEO compliant.

Better CMS

My work-flow is not the most efficient. Each article points to an actual file, which is not so good, because I need to create a file each I add an entry in this blog. I will improve this soon. It is one of the reason why I stopped writing here. It is too complex.

Migrating the last virtual from Xen to LXD

I need to migrate completely my photo galleries (Piwigo) from XEN to the new ones on LXD.

Home made modem/router/NAS

I bought few items in order to build my own modem/router/NAS.


Although RSS feed appears to work fine, there are PHP errors in the logs. I need to investigate.

Dr Clément Février

Bonjour, Je suis Clément Février, docteur en physique théorique de l’université de Grenoble Alpes, ingénieur Recherche et Développement dans le domaine de l’imagerie médicale et de la chirurgie mini-invasive chez Surgivisio et soutien du mouvement La France Insoumise.

Dites les velotaffeur, vous avez entendu parler d'un casque qui a un feu avant, un feu arrière et des clignotants sur les cotés / avant arrière ? (Enfin un simple bandeau led de chaque coté se gère hein)
Genre un truc pour que les voitures voient qu'on tourne à droite ou gauche sans qu'on ai besoin de tendre le bras ?

Parce que bon, j'ai testé ce soir, passer sur les marquages au sol + les rails du tramway avec un bras levé sous la pluie, niveau stabilité on repassera ^^'

J'arrive pas à trouver :/ et les solutions existantes de clignotants sont seulement pour l'arrière

Si le coeur vous en dit vous pouvez partager ;)

#velotaff #boostappréciés :)

The only thing that really matter is, at the end of the day, to choose the correct method automatically without conditional statement at runtime. I really wonder is a workaround is possible.
I already tried to add "std::tuple< d1*, d2 > b_tuple" to class b, using some forward declaration, but, of course, std::get< i >(b_tuple) cannot compile since it will be known at runtime only.

But I allow complete refactoring code, even the b and d1 & d2. Templates, new classes and sub classes and even are fine (I already put 100-lines define to add iterator to enum and another one to allow enum inheritance, and my coworkers begin to hate me ^^)
For example, method "create" can become a class, or class b can be construct using a macro to make somehow the base class aware of derived ones
BASE(b, ...) std::tuple< __VA_ARGS__ > g_ ## b; class b

How to choose a function at run time using overloaded resolution in ?
I feel it's an old known issue without (obvious) answer, but many things changed since c++98

I add the new following constrain to my original question, it can be up to C++14, no more (although if a solution in c++17 or C++20 exist, I'm interested) because of compiler limitation (appears to be based on gcc 5.14 from what I understood).