Instance boot error and I can't connect

Google Cloud Compute Infrastructure
1

I need help!
My instance apparently starts. But I can’t connect.
The log displays a disk full message. I increased disk space and still can’t connect. Serial port output returns the message below.

Ubuntu 21.04 bancodedados ttyS0

bancodedados login: [  549.942514] cloud-init[987]: OSError: [Errno 28] No space left on device
[  549.942724] cloud-init[987]: During handling of the above exception, another exception occurred:
[  549.942816] cloud-init[987]: Traceback (most recent call last):
[  549.942875] cloud-init[987]:   File "/usr/bin/cloud-init", line 33, in <module>
[  549.942941] cloud-init[987]:     sys.exit(load_entry_point('cloud-init==21.4', 'console_scripts', 'cloud-init')())
[  549.942986] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
[  549.943035] cloud-init[987]:     retval = util.log_time(
[  549.943078] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
[  549.943416] cloud-init[987]:     ret = func(*args, **kwargs)
[  549.943500] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 684, in status_wrapper
[  549.943849] cloud-init[987]:     atomic_helper.write_json(status_path, status)
[  549.943930] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 44, in write_json
[  549.944082] cloud-init[987]:     return write_file(
[  549.944133] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 39, in write_file
[  549.944201] cloud-init[987]:     raise e
[  549.944250] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 33, in write_file
[  549.944295] cloud-init[987]:     tf.close()
[  549.944337] cloud-init[987]:   File "/usr/lib/python3.9/tempfile.py", line 639, in close
[  549.944613] cloud-init[987]:     self._closer.close()
[  549.944685] cloud-init[987]:   File "/usr/lib/python3.9/tempfile.py", line 573, in close
[  549.944976] cloud-init[987]:     self.file.close()
[  549.945063] cloud-init[987]: OSError: [Errno 28] No space left on device
[  549.945114] cloud-init[987]: Error in sys.excepthook:
[  549.945166] cloud-init[987]: Traceback (most recent call last):
[  549.945213] cloud-init[987]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 155, in apport_excepthook
[  549.945265] cloud-init[987]:     pr.write(f)
[  549.945308] cloud-init[987]:   File "/usr/lib/python3/dist-packages/problem_report.py", line 382, in write
[  549.945362] cloud-init[987]:     file.write(v.replace(b'\n', b'\n '))
[  549.945535] cloud-init[987]: OSError: [Errno 28] No space left on device
[  549.945587] cloud-init[987]: During handling of the above exception, another exception occurred:
[  549.945629] cloud-init[987]: Traceback (most recent call last):
[  549.945681] cloud-init[987]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 155, in apport_excepthook
[  549.945744] cloud-init[987]:     pr.write(f)
[  549.945790] cloud-init[987]: OSError: [Errno 28] No space left on device
[  549.945917] cloud-init[987]: Original exception was:
[  549.945959] cloud-init[987]: OSError: [Errno 28] No space left on device
[  549.946001] cloud-init[987]: During handling of the above exception, another exception occurred:
[  549.946045] cloud-init[987]: Traceback (most recent call last):
[  549.946087] cloud-init[987]:   File "/usr/bin/cloud-init", line 33, in <module>
[  549.946188] cloud-init[987]:     sys.exit(load_entry_point('cloud-init==21.4', 'console_scripts', 'cloud-init')())
[  549.946244] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
[  549.946357] cloud-init[987]:     retval = util.log_time(
[  549.946403] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
[  549.946931] cloud-init[987]:     ret = func(*args, **kwargs)
[  549.947011] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 684, in status_wrapper
[  549.947324] cloud-init[987]:     atomic_helper.write_json(status_path, status)
[  549.947392] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 44, in write_json
[  549.947452] cloud-init[987]:     return write_file(
[  549.947501] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 39, in write_file
[  549.947739] cloud-init[987]:     raise e
[  550.248548] cloud-init[987]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 33, in write_file
[  550.248722] cloud-init[987]:     tf.close()
[  550.248772] cloud-init[987]:   File "/usr/lib/python3.9/tempfile.py", line 639, in close
[  550.248805] cloud-init[987]:     self._closer.close()
[  550.248834] cloud-init[987]:   File "/usr/lib/python3.9/tempfile.py", line 573, in close
[  550.248863] cloud-init[987]:     self.file.close()
[  550.248897] cloud-init[987]: OSError: [Errno 28] No space left on device
[  551.380242] cloud-init[988]: OSError: [Errno 28] No space left on device
[  551.380549] cloud-init[988]: During handling of the above exception, another exception occurred:
[  551.380606] cloud-init[988]: Traceback (most recent call last):
[  551.380651] cloud-init[988]:   File "/usr/bin/cloud-init", line 33, in <module>
[  551.380712] cloud-init[988]:     sys.exit(load_entry_point('cloud-init==21.4', 'console_scripts', 'cloud-init')())
[  551.380757] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
[  551.380810] cloud-init[988]:     retval = util.log_time(
[  551.380856] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
[  551.380910] cloud-init[988]:     ret = func(*args, **kwargs)
[  551.380953] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 684, in status_wrapper
[  551.382313] cloud-init[988]:     atomic_helper.write_json(status_path, status)
[  551.382407] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 44, in write_json
[  551.382454] cloud-init[988]:     return write_file(
[  551.382510] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 39, in write_file
[  551.382558] cloud-init[988]:     raise e
[  551.382603] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 33, in write_file
[  551.382646] cloud-init[988]:     tf.close()
[  551.382695] cloud-init[988]:   File "/usr/lib/python3.9/tempfile.py", line 639, in close
[  551.382737] cloud-init[988]:     self._closer.close()
[  551.382779] cloud-init[988]:   File "/usr/lib/python3.9/tempfile.py", line 573, in close
[  551.382821] cloud-init[988]:     self.file.close()
[  551.382863] cloud-init[988]: OSError: [Errno 28] No space left on device
[  551.382904] cloud-init[988]: Error in sys.excepthook:
[  551.382946] cloud-init[988]: Traceback (most recent call last):
[  551.382987] cloud-init[988]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 155, in apport_excepthook
[  551.383028] cloud-init[988]:     pr.write(f)
[  551.383070] cloud-init[988]:   File "/usr/lib/python3/dist-packages/problem_report.py", line 382, in write
[  551.383115] cloud-init[988]:     file.write(v.replace(b'\n', b'\n '))
[  551.383159] cloud-init[988]: OSError: [Errno 28] No space left on device
[  551.383203] cloud-init[988]: During handling of the above exception, another exception occurred:
[  551.383381] cloud-init[988]: Traceback (most recent call last):
[  551.383435] cloud-init[988]:   File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 155, in apport_excepthook
[  551.383490] cloud-init[988]:     pr.write(f)
[  551.383536] cloud-init[988]: OSError: [Errno 28] No space left on device
[  551.383582] cloud-init[988]: Original exception was:
[  551.383625] cloud-init[988]: OSError: [Errno 28] No space left on device
[  551.383667] cloud-init[988]: During handling of the above exception, another exception occurred:
[  551.383715] cloud-init[988]: Traceback (most recent call last):
[  551.383756] cloud-init[988]:   File "/usr/bin/cloud-init", line 33, in <module>
[  551.383803] cloud-init[988]:     sys.exit(load_entry_point('cloud-init==21.4', 'console_scripts', 'cloud-init')())
[  551.383853] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
[  551.383919] cloud-init[988]:     retval = util.log_time(
[  551.383965] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
[  551.384005] cloud-init[988]:     ret = func(*args, **kwargs)
[  551.384044] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 684, in status_wrapper
[  551.384088] cloud-init[988]:     atomic_helper.write_json(status_path, status)
[  551.384133] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 44, in write_json
[  551.384182] cloud-init[988]:     return write_file(
[  551.384226] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 39, in write_file
[  551.727184] cloud-init[988]:     raise e
[  551.727254] cloud-init[988]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 33, in write_file
[  551.727289] cloud-init[988]:     tf.close()
[  551.727319] cloud-init[988]:   File "/usr/lib/python3.9/tempfile.py", line 639, in close
[  551.727347] cloud-init[988]:     self._closer.close()
[  551.727376] cloud-init[988]:   File "/usr/lib/python3.9/tempfile.py", line 573, in close
[  551.728142] cloud-init[988]:     self.file.close()
[  551.728180] cloud-init[988]: OSError: [Errno 28] No space left on device
[  610.581427] google_guest_agent[558]: 2022/03/11 09:59:40 logging client: context deadline exceeded
[ 1210.581652] google_guest_agent[558]: 2022/03/11 10:09:40 logging client: context deadline exceeded
[ 1810.581942] google_guest_agent[558]: 2022/03/11 10:19:40 logging client: context deadline exceeded
[ 2410.582202] google_guest_agent[558]: 2022/03/11 10:29:40 logging client: context deadline exceeded
[ 3010.582585] google_guest_agent[558]: 2022/03/11 10:39:40 logging client: context deadline exceeded
[ 3610.582876] google_guest_agent[558]: 2022/03/11 10:49:40 logging client: context deadline exceeded
2

You might have to resize the file system manually. If your VM does not support automatic root file system resizing, or if you’ve resized a data (non-boot) persistent disk, you must manually resize the file system and partitions.

I also recommend you run command “df -i” to check the file system, and clean up “/tmp”.

3

I have the same issue and I cannot even SSH to it.

How will I be able to run those commands?

4

I believe the solution that remains is to attach the disk to another temp created instance, remove unneeded files and retry attaching then working on the same old machine

5

I resolved the problem in a simple way.
I did use the steps described in the documentation and the system was restored

https://cloud.google.com/compute/docs/disks/recover-vm

This other link may also help

https://cloud.google.com/compute/docs/disks/resize-persistent-disk