There can be various causes why a Linux VPS no longer boots and displays an error message. In most cases this is due to a misconfiguration of your VPS that often only becomes apparent later, for example after a VPS restart. These are often quite technical problems that can be complicated to resolve. Restoring a backup is not always desirable or possible.
In this guide we present a range of such possible problems and explain how to resolve them.
- Create a snapshot of your VPS before you begin the steps in this guide so that in case of emergency you have a copy to fall back on.
- CentOS Stream / AlmaLinux / Rocky Linux: In some cases, SELinux must undergo a so-called “relabel” process afterward. After completing the steps in this article, use the command journalctl to check if there is a message about it. If so, then use the command shown in that message (usually sudo touch/.autorelabel; reboot).
error: file '/vmlinuz-<version>-generic' not found. error: you need to load a kernel first
As the error message indicates, the kernel cannot be found. Normally the kernel is located in the /boot/ directory.
Before proceeding, note down the version number shown in the error message—for example, 6.8.0-54 in the message ‘error: file '/vmlinuz-6.8.0-54-generic' not found.’
In CentOS Stream, AlmaLinux, and Rocky Linux you will see a bit more text than in the example above, but in any case note the kernel version—for example: vmlinuz-5.14.0-503.26.1.el9_5.x86_64 (with a lowercase L and not a 1 after the e).
Step 1
In the TransIP control panel, go to the VPS in question. In the VPS console, click the pop-out button at the bottom left:
Then click on ‘Rescue Linux’.
It takes a few minutes for Rescue mode to start, and it will look as follows:
Step 2
First, check which block devices are available:
lsblkThe output will look roughly as follows (usually with a vda1 and vda2 on Red Hat systems):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 100G 0 disk
├─vda1 253:1 0 99G 0 part /
├─vda14 253:14 0 4M 0 part
├─vda15 253:15 0 106M 0 part /boot/efi
└─vda16 259:0 0 913M 0 part /bootUbuntu/Debian
Mount the first device you see under ‘vda’: in this example, that is vda1:
mount /dev/vda1 /mntCentOS Stream/AlmaLinux/Rocky Linux
Mount vda2 (your root filesystem) and vda1 (your boot partition):
mount /dev/vda2 /mnt
mkdir /mnt/boot
mount /dev/vda1 /mnt/boot
Step 3
Mount dev, proc, sys, and tmp so that you can use them from a chroot environment (chroot stands for “change root” and changes the root directory):
mount --rbind /dev /mnt/dev
mount --make-rslave /mnt/dev
mount -t proc /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys
mount --rbind /tmp /mnt/tmp
Step 4
Now chroot into the mounted filesystem:
chroot /mnt /bin/bash
Step 5
Reinstall Grub and update the Grub configuration:
Ubuntu / Debian
grub-install /dev/vda
update-grubCentOS Stream / AlmaLinux / Rocky Linux
- Replace
/dev/vda2with the name of the block device containing your filesystem (not the boot partition), see Step 2: - Note that you must use double quotes/brackets in the commands in this section. When pasting into the VPS console, these are likely to be automatically changed to single quotes/brackets.
Now execute the following commands (pay attention to the brackets):
UUID=$(blkid /dev/vda2 -o value -s UUID)
KERNELCONF=$(basename "$(ls /boot/loader/entries/*.conf | sort | tail -n 1)" .conf)
grub2-editenv - set "saved_entry=$KERNELCONF"
grub2-editenv - set "boot_succes=0"
grub2-editenv - set "boot_indeterminate=0"
grub2-editenv - set "kernelopts=root=UUID=$UUID ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M net.ifnames=0 rhgb quiet"
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-install /dev/vda
grub2-mkconfig -o /boot/grub2/grub.cfgOptionally, you can check the value of the kernelopts and options variables as follows:
cat /boot/loader/entries/$KERNELCONF | grep options
grub2-editenv listExplanation
For a successful boot, the variable kernelopts must exist, which is generally not the case when you go through this process.
A typo is easily made, so you create a few variables that contain the UUID and filename of the kernel configuration. The commands above do the following:
- First, you create a variable $UUID that holds the UUID of your block device
- Next, you create a variable that contains the filename of your kernel configuration
- Then, you set up the necessary Grub environment variables
- Finally, you update the Grub configuration, reinstall Grub, and update the Grub configuration once again
Step 6
Now exit the chroot environment:
exit
Step 7
Unmount all mounted folders. The ‘-l’ option stands for “lazy” and forces unmounting even if you would otherwise receive a “busy” error:
Ubuntu / Debian
umount -l /mnt/tmp
umount -l /mnt/sys
umount -l /mnt/proc
umount -l /mnt/dev
umount -l /mntCentOS Stream / AlmaLinux / Rocky Linux
umount -l /mnt/tmp
umount -l /mnt/sys
umount -l /mnt/proc
umount -l /mnt/dev
umount -l /mnt/boot
umount -l /mnt
Step 8
Restart your VPS via the ‘Restart’ button in the VPS console: a reboot command will not work because it takes you to the PXE environment.
Congratulations! Your VPS should now boot normally again (this may take a while, and you might see a few error messages that your system automatically resolves during startup).
Welcome to GRUB!
A friendly message, but not the one you expected. The full message looks roughly as follows:
Booting from Hard Hisk...
GRUB loading...
Welcome to GRUB!
error: ../.. <error message>
Entering rescue mode...
grub rescue>This message simply indicates that something is wrong with GRUB, for example that the GRUB configuration has been deleted or contains an error, and your operating system cannot boot.
Step 1
In the TransIP control panel, go to the VPS in question. In the VPS console, click the pop-out button at the bottom left:
Then click on ‘Rescue Linux’.
It takes a few minutes for Rescue mode to start, and it will look as follows:
Step 2
First, check which block devices are available:
lsblkThe output will look roughly as follows (usually with a vda1 and vda2 on Red Hat systems):
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 253:0 0 100G 0 disk
├─vda1 253:1 0 99G 0 part /
├─vda14 253:14 0 4M 0 part
├─vda15 253:15 0 106M 0 part /boot/efi
└─vda16 259:0 0 913M 0 part /bootUbuntu/Debian
Mount the first device you see under ‘vda’: in this example, that is vda1:
mount /dev/vda1 /mntCentOS Stream/AlmaLinux/Rocky Linux
Mount vda2 (your root filesystem) and vda1 (your boot partition):
mount /dev/vda2 /mnt
mkdir /mnt/boot
mount /dev/vda1 /mnt/boot
Step 3
Mount dev, proc, sys, and tmp so that you can use them from a chroot environment (chroot stands for “change root” and changes the root directory):
mount --rbind /dev /mnt/dev
mount --make-rslave /mnt/dev
mount -t proc /proc /mnt/proc
mount --rbind /sys /mnt/sys
mount --make-rslave /mnt/sys
mount --rbind /tmp /mnt/tmp
Step 4
Now chroot into the mounted filesystem:
chroot /mnt /bin/bash
Step 5
Reinstall Grub and update the Grub configuration:
Ubuntu / Debian
grub-install /dev/vda
update-grubCentOS Stream / AlmaLinux / Rocky Linux
- Replace
/dev/vda2with the name of the block device containing your filesystem (not the boot partition), see Step 2: - Note that you must use double brackets in the commands in this section. When pasting into the VPS console, these will probably be automatically changed to single brackets.
Now execute the following commands (pay attention to the brackets):
UUID=$(blkid /dev/vda2 -o value -s UUID)
KERNELCONF=$(basename "$(ls /boot/loader/entries/*.conf | sort | tail -n 1)" .conf)
grub2-editenv - set "saved_entry=$KERNELCONF"
grub2-editenv - set "boot_succes=0"
grub2-editenv - set "boot_indeterminate=0"
grub2-editenv - set "kernelopts=root=UUID=$UUID ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M net.ifnames=0 rhgb quiet"
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-install /dev/vda
grub2-mkconfig -o /boot/grub2/grub.cfgExplanation
For a successful boot, the variable kernelopts must exist, which is generally not the case when you go through this process.
A typo is easily made, so you create a few variables that contain the UUID and filename of the kernel configuration. The commands above do the following:
- First, you create a variable $UUID that holds the UUID of your block device
- Next, you create a variable that contains the filename of your kernel configuration
- Then, you set up the necessary Grub environment variables
- Finally, you update the Grub configuration, reinstall Grub, and update the Grub configuration once again
Step 6
Now exit the chroot environment:
exit
Step 7
Unmount all mounted folders. The ‘-l’ option stands for “lazy” and forces the unmounting even if you would otherwise receive a “busy” error:
Ubuntu / Debian
umount -l /mnt/tmp
umount -l /mnt/sys
umount -l /mnt/proc
umount -l /mnt/dev
umount -l /mntCentOS Stream / AlmaLinux / Rocky Linux
umount -l /mnt/tmp
umount -l /mnt/sys
umount -l /mnt/proc
umount -l /mnt/dev
umount -l /mnt/boot
umount -l /mnt
Step 8
Restart your VPS via the ‘Restart’ button in the VPS console: a reboot command will not work because it takes you to the PXE environment.
Congratulations! Your VPS should now boot normally again (this may take a while and you might see some error messages that your system automatically resolves during startup).
Probably out of disk space / <service>.service: Failed with the result ‘exit-code’.
Such an error message will not prevent your VPS from booting, but it may prevent a service such as MariaDB from starting. During the boot process, you might already see a red message indicating that the respective service cannot start. There may be several causes, but in this case we assume it is due to a lack of disk space. The steps below can also be used (partially) for other causes.
Step 1
When a service does not start, a good starting point is to check the associated log files. You can do this with the following command, replacing <service> with the name of the service (for example, mariadb, apache2, or httpd):
journalctl -xe -u <service>
Step 2
You will probably see the necessary text, but the most relevant lines are those containing “error”, for example in this MariaDB example:
The job identifier is 735.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] Starting MariaDB 11.4.5-MariaDB source revision 0771110266ff5c04216af4bf1243c65f8c67ccf4 server_uid CtPOHqBTdn3uD4>
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: Number of transaction pools: 1
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] InnoDB: preallocating 12582912 bytes for file ./ibtmp1 failed with error 28
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: FTS optimize thread exiting.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: Starting shutdown...
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] Plugin 'FEEDBACK' is disabled.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [Note] Plugin 'wsrep-provider' is disabled.
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Mar 06 14:54:34 example.transip.nl mariadbd[1428]: 2025-03-06 14:54:34 0 [ERROR] Aborting
Mar 06 14:54:34 example.transip.nl systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ An ExecStart= process belonging to unit mariadb.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Mar 06 14:54:34 example.transip.nl systemd[1]: mariadb.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
Mar 06 14:54:34 example.transip.nl systemd[1]: Failed to start MariaDB 11.4.5 database server.
░░ Subject: A start job for unit mariadb.service has failed
░░ Defined-By: systemd
░░ Support: https://wiki.almalinux.org/Help-and-Support
░░
░░ A start job for unit mariadb.service has finished with a failure.
░░
░░ The job identifier is 735 and the job result is failed.You can ignore the lower part: the relevant messages are the lines with [ERROR]. In particular, the following message is most relevant:
Could not set the file size of './ibtmp1'. Probably out of disk spaceNow press ‘q’ to close journalctl.
Step 3
It is quite simple to check whether there is indeed too little disk space. Use the following command:
df -hThe output will look roughly as follows:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 732M 8.6M 724M 2% /run
/dev/vda2 147G 139G 12M 100% /
/dev/vda1 504M 480M 0 100% /boot
tmpfs 366M 0 366M 0% /run/user/1000As you can see, there is no space in the boot partition and almost no space in the filesystem (/dev/vda2). The next step is mainly to determine what exactly is taking up so much space in both.
Step 4
In the case of the boot partition, there is a good chance that you still have a number of old kernels stored. Fortunately, these can be easily removed as follows:
Ubuntu / Debian
sudo apt -y autoremove --purgeTab Title
package-cleanup --oldkernels --count=2With this command, the old kernels are removed and the newest two are kept.
If you receive an error message that the required software is not installed, you cannot install it as long as there is no free disk space. In that case, use the command:
dnf remove $(dnf repoquery --installonly --latest-limit=-2 -q)For your filesystem, there is also a simple option to determine what takes up the most space. Use the following command:
du -ah / 2>/dev/null | sort -rh | head -n 20The output will look roughly as follows:
127G /
118G /tmp
118G /tmp/fullfile
3.2G /usr
3.1G /data/registry/docker/registry/v2/blobs/sha256
3.1G /data/registry/docker/registry/v2/blobs
3.1G /data/registry/docker/registry/v2
3.1G /data/registry/docker/registry
3.1G /data/registry/docker
3.1G /data/registry
3.1G /data
1.5G /home/transip
1.5G /home
1.5G /data/registry/docker/registry/v2/blobs/sha256/68/685403537ceac572aee2828c05945053e897cce03362f0b89d2a00931f18c141/data
1.5G /data/registry/docker/registry/v2/blobs/sha256/68/685403537ceac572aee2828c05945053e897cce03362f0b89d2a00931f18c141
1.5G /data/registry/docker/registry/v2/blobs/sha256/68
1.5G /data/registry/docker/registry/v2/blobs/sha256/5c/5c42024024cb80a8c37e678ad3f533112343e91ebd87e59d1796476b71435474/data
1.5G /data/registry/docker/registry/v2/blobs/sha256/5c/5c42024024cb80a8c37e678ad3f533112343e91ebd87e59d1796476b71435474
1.5G /data/registry/docker/registry/v2/blobs/sha256/5cAs you can see, the /tmp folder contains 118G of data and there is a file called fullfile in the /tmp folder that is a whopping 118G. This is a test file that can be easily removed:
sudo rm /tmp/fullfileIf you wish to remove entire directories, use the command:
sudo rm -rf <folder>Be careful and only do this for custom folders such as /home/<user>/<somefolder>, never for system folders such as /boot/ or /etc/. Replace <folder> with the path of the directory you wish to remove, for example:
sudo rm -rf /home/transip/example/
Step 5
Restart the service that has stalled due to the lack of disk space, or better yet, your entire VPS so that you can be sure all services are available again:
sudo systemctl restart <service>
sudo reboot