Jump to content

User:Razzi/First logical volume resizing

From Wikitech
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
puppet $ sshrc an-test-coord1001.eqiad.wmnet
Linux an-test-coord1001 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
Debian GNU/Linux 10 (buster)

  _  __         _               _             _   _               _
 | |/ /        | |             (_)           | | | |             | |
 | ' / ___ _ __| |__   ___ _ __ _ _______  __| | | |__   ___  ___| |_
 |  < / _ \ '__| '_ \ / _ \ '__| |_  / _ \/ _` | | '_ \ / _ \/ __| __|
 | . \  __/ |  | |_) |  __/ |  | |/ /  __/ (_| | | | | | (_) \__ \ |_
 |_|\_\___|_|  |_.__/ \___|_|  |_/___\___|\__,_| |_| |_|\___/|___/\__|


This host is capable of Kerberos authentication in the WIKIMEDIA realm.

For more info: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide

an-test-coord1001 is a Analytics Cluster host running various Hadoop services (Hive, Camus, Oozie, ..) and maintenance scripts (analytics_test_cluster::coordinator)
Debian GNU/Linux 10 auto-installed on Fri Apr 2 14:23:58 UTC 2021.
Last login: Mon Jan 24 16:57:47 2022 from 2620:0:863:1:198

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
welcome
razzi@an-test-coord1001:~$ ls
apache-atlas-2.2.0-sources.tar  cookiejar.txt  pidinfo.txt
apache-atlas-sources-2.2.0      index.html     run_atlas
razzi@an-test-coord1001:~$ df -h
df: /mnt/hdfs: Input/output error
Filesystem            Size  Used Avail Use% Mounted on
udev                   63G     0   63G   0% /dev
tmpfs                  13G  1.4G   12G  11% /run
/dev/mapper/vg0-root   73G   50G   20G  72% /
tmpfs                  63G     0   63G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/vg0-srv   365G   29G  318G   9% /srv
tmpfs                  13G     0   13G   0% /run/user/0
tmpfs                  13G     0   13G   0% /run/user/118
tmpfs                  13G     0   13G   0% /run/user/32741
tmpfs                  13G     0   13G   0% /run/user/2543
tmpfs                  13G     0   13G   0% /run/user/26051
razzi@an-test-coord1001:~$ man lvreduce
razzi@an-test-coord1001:~$ sudo lvs
  LV   VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root vg0 -wi-ao----   74.50g
  srv  vg0 -wi-ao---- <371.27g
  swap vg0 -wi-ao----  976.00m
razzi@an-test-coord1001:~$ lvreduce srv -L -2
bash: lvreduce: command not found
razzi@an-test-coord1001:~$ sudo lvreduce srv -L -2
  Please specify a logical volume path.
  Run `lvreduce --help' for more information.
razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-srv -L -2G
  WARNING: Reducing active and open logical volume to <369.27 GiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce vg0/srv? [y/n]: y
  Size of logical volume vg0/srv changed from <371.27 GiB (95044 extents) to <369.27 GiB (94532 extents).
  Logical volume vg0/srv successfully resized.
razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-root -L +2G
  Size may not be positive.
  Invalid argument for --size: +2G
  Error during parsing of command line.
razzi@an-test-coord1001:~$ sudo lvextend /dev/mapper/vg0-root -L+2G
  Size of logical volume vg0/root changed from 74.50 GiB (19073 extents) to 76.50 GiB (19585 extents).
  Logical volume vg0/root successfully resized.
razzi@an-test-coord1001:~$ resize2fs /dev/mapper/vg0-root
bash: resize2fs: command not found
razzi@an-test-coord1001:~$ sudo resize2fs /dev/mapper/vg0-root
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mapper/vg0-root is mounted on /; on-line resizing required
old_desc_blocks = 10, new_desc_blocks = 10
The filesystem on /dev/mapper/vg0-root is now 20055040 (4k) blocks long.

razzi@an-test-coord1001:~$ df -h
df: /mnt/hdfs: Input/output error
Filesystem            Size  Used Avail Use% Mounted on
udev                   63G     0   63G   0% /dev
tmpfs                  13G  1.4G   12G  11% /run
/dev/mapper/vg0-root   75G   50G   22G  70% /
tmpfs                  63G     0   63G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/vg0-srv   365G   29G  318G   9% /srv
tmpfs                  13G     0   13G   0% /run/user/0
tmpfs                  13G     0   13G   0% /run/user/118
tmpfs                  13G     0   13G   0% /run/user/32741
tmpfs                  13G     0   13G   0% /run/user/2543
tmpfs                  13G     0   13G   0% /run/user/26051

I did the above in the wrong order; we were fortunately able to create a backup of mariadb from the corrupted volume and restore that to srv. Some other folders were manually recreated and some were created from a puppet run. The puppet run errored however, with a systemd issue: 

Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ...
 systemd[1]: Caught <ABRT>, dumped core as pid 15168.

Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC):

systemd[1]: Caught <ABRT>, dumped core as pid 15168.


Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ...
 systemd[1]: Freezing execution.

Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC):

systemd[1]: Freezing execution.


razzi@an-test-coord1001:~$
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Razzi/First_logical_volume_resizing&oldid=1942923"