Categories
FreeBSD GitLab SSH

GitLab: SSH key does not work

After setting up GitLab server I found git operations via SSH did not work. I tried the following:

  1. rake gitlab:check to check for any obvious errors
  2. rake gitlab:shell:setup as suggested in this ticket, there is another more recently opened ticket
  3. regenerate SSH client keys
  4. checking the permissions of /usr/local/git/.ssh and /home/git/.ssh for git user

Everything from the above seemed to check out.

However the root cause in my case turned out to be that the location of the authorized_keys file for git user needed to be updated in the GitLab YAML configuration file gitlab.yml from:

  ## GitLab Shell settings
  gitlab_shell:
    path: /usr/local/share/gitlab-shell/
    authorized_keys_file: /usr/local/git/.ssh/authorized_keys

to:

  ## GitLab Shell settings
  gitlab_shell:
    path: /usr/local/share/gitlab-shell/
    authorized_keys_file: /home/git/.ssh/authorized_keys

Then after re-running rake gitlab:shell:setup a git user’s SSH started working.

This appeared to be the FreeBSD port specific issue where gitlab.yml.example was patched replacing /home with /usr/local prefix for paths in the YAML configuration file.

Categories
*BSD UPS

UPS Monitoring With NUT On FreeBSD

Some uninterruptible power supplies offer a serial interface. The Network UPS Tools project provides an ability to interact with such devices, usually via USB. On FreeBSD NUT is available as a port and it is possible to install from the package manager:

pkg info nut
nut-2.8.0_16
Name           : nut
Version        : 2.8.0_16
Installed on   : Fri Jan 27 18:24:41 2023 EST
Origin         : sysutils/nut
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : GPLv3+ and GPLv2+ and GPLv1+ and ART10
Maintainer     : cy@FreeBSD.org
WWW            : https://www.networkupstools.org/
Comment        : Network UPS Tools
sudo pkg install nut

In my case I have the CyberPower EC750G device so the configuration /usr/local/etc/nut/ups.conf looks as follows:

[CPS_EC750G]
    driver = usbhid-ups
    port = auto

It is also required to set devfs rules for the UPS device. Upon latest installation NUT runs under nut:nut user:group. The user:group is for the port installation from GhostBSD. The user:group for the port from FreeBSD is uucp:uucp. To find USB serial connection one run the following command:

sudo usbconfig
ugen0.17: <CPS EC750G> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (50mA)

Given the above information we can configure the devfs rules as follows:

/etc/devfs.conf
perm    ugen0.17        0770
own     ugen0.17        nut:nut

/etc/devfs.rules
nut_usb=10]
  add path 'ugen0.17' group nut user nut mode 0770
  add path 'usb/0.17.0' group nut user nut mode 0770

Now we can configure NUT service as follows

sudo sysrc devfs_system_rules="nut_usb"
sudo sysrc nut_enable=YES
sudo sysrc upslog_enable=YES
sudo sysrc upsmon_enable=YES

Finally we can start up NUT and check with UPS client we can connect to our device

sudo service restart nut

upsc -l
CPS_EC750G

By default NUT listens binds to local loopback interface and listens on port 3493. This is configurable in /usr/local/etc/nut/upsd.conf

netstat -an|grep tcp|grep 3493
tcp6       0      0 ::1.3493               *.*                    LISTEN     
tcp4       0      0 127.0.0.1.3493         *.*                    LISTEN

Additionally there is also nut_exporter available in the FreeBSD port collection if one would like to add UPS metrics to Prometheus and consequently to Grafana in order to visualize battery capacity, power usage, etc.

pkg info nut_exporter
nut_exporter-2.5.2
Name           : nut_exporter
Version        : 2.5.2
Installed on   : Sat Jan 28 14:12:44 2023 EST
Origin         : sysutils/nut_exporter
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : APACHE20
Maintainer     : bsd@j-star.org
WWW            : https://github.com/DRuggeri/nut_exporter
Comment        : Prometheus exporter for the Network UPS Tools server

This exporter listens on port 9199 and makes NUT metrics available via /ups_metrics URL path.

The Prometheus YAML configuration to add the NUT metrics looks as follows:

- job_name: "nut"
  static_configs:
       - targets: ["localhost:9199"] 
  metrics_path: /ups_metrics
Categories
*BSD Embedded

Manually Booting Board Via U-Boot

Many micro-controllers or single-board computers use U-Boot system to boot the loader. Sometimes it may happen that such a board fails to boot. If the board in question uses U-Boot it should be possible to interrupt the failing boot process in order to enter U-Boot command-line interface by pressing any key as the boot-up prompt suggests. Once we are in the U-Boot command line we can interact with it. To print out list of commands we can enter either ‘?’ or ‘help’.

If the boot-up error is due to failure finding and loading the system loader we can use the U-Boot command-line interface to tell it where to look. The following will set boot command with the loader device pointing to disk 0, slice 3a and loading the system loader binary ubldr.bin (FreeBSD part of loader utilizing U-Boot)

set bootcmd "set loaderdev disk0s3a; fatload mmc 0 $kernel_addr_r ubldr.bin ; go $kernel_addr_r"

If U-Boot fails to find a device tree blob for the board we tell it which one to use via the following command (device tree blob for BeagleBone Green Wireless)

set fdtfile  am335x-bonegreen-wireless.dtb

Next assuming the loader device is correct and the system loader and the device tree blob exist we can resume booting 

boot
Categories
*BSD ZFS

Importing ZFS Data Pool From Older FreeBSD System

CAUTION: after the following instructions are executed the older system will become unbootable. Please ensure the data is backed up

If we have two FreeBSD systems with ZFS and would like to access data from one on the other we will need to import the ZFS pool in such the way that mount points from the older system do not collide with the new system.

First we will need to import the pool without mounting datasets

sudo zpool import -N tank

Next we will need to ensure the pool unsets mount points for various system file locations to prevent collisions of mount points from two pools

sudo zfs set mountpoint=none tank
sudo zfs set mountpoint=none tank/usr/home
sudo zfs set mountpoint=none tank/usr/jails
sudo zfs set mountpoint=none tank/usr/local
sudo zfs set mountpoint=none tank/usr/obj
sudo zfs set mountpoint=none tank/usr/src
sudo zfs set mountpoint=none tank/usr/ports
sudo zfs set mountpoint=none tank/tmp
sudo zfs set mountpoint=none tank/var
sudo zfs set mountpoint=none tank/var/audit
sudo zfs set mountpoint=none tank/var/log
sudo zfs set mountpoint=none tank/var/mail
sudo zfs set mountpoint=none tank/var/tmp

We will set up the file location where we would like to mount data from the imported pool and configure the pool mount points accordingly

sudo mkdir -p /export/usr/home/max
sudo zfs set mountpoint=/export tank/ROOT/12.0-CURRENT
sudo zfs set mountpoint=/export/usr tank/usr
sudo zfs set mountpoint=/export/usr/home/max tank/usr/home/max

We will need to double check that the pool mount points are set up correctly and ensure they are either set to none or to our non-system mount points

zfs list |grep tank

Once above looks good let us export and re-import the pool and ensure there are no failures or errors when mounting the pool

sudo zpool export tank
sudo zpool import tank

Lastly we will need to reboot the system and check that both pools are imported and mounted

zpool list -o name,health
NAME       HEALTH
tank       ONLINE
trident    ONLINE

zfs mount |grep tank
Categories
Databases

Migrating From MySQL 8 to MariaDB

CAUTION Please ensure backing up a MySQL database directory before proceeding with the below steps.

There is a couple of things to change when migrating MySQL server configuration and databases to MariaDB.

First on startup MariaDB fails to come up due to unsupported configuration settings from the MySQL server my.cnf. The following settings need to be removed:

enforce-gtid-consistency       = 1
gtid-mode                       = ON

Second importing MySQL 8 databases into MariaDB may fail with the following reason:

Unknown collation: utf8mb4_0900_ai_ci

Please ensure MySQL 8 database dump is backed up. After replacing utf8mb4_0900_ai_ci to utf8mb4_unicode_520_ci MySQL 8 database import into MariaDB runs successfully.

The last thing that probably needs updating is reinstating privileges to database users:

grant all privileges on `database-user`@`host-from-which-user access-database` identified by 'database-user-password';
flush privileges;
Categories
Databases

Recover MySQL Database After Page Corruption

When a server outage happens MySQL may not shut down cleanly and upon restart it may fail with the following error:

2022-07-09T01:29:38.886478Z 0 [ERROR] InnoDB: Database page corruption on disk or a failed file read of page [page id: space=0, page number=505]. You may have to recover from a backup.

CAUTION Please ensure backing up a MySQL database directory before proceeding with the below steps.

First we need to start MySQL in recovery mode by the setting the following configuration in /usr/local/etc/mysql/my.cnf:

innodb_force_recovery           = 1

Once MySQL is up we can dump MySQL databases using following command:

mysqldump -u root -p --all-databases --triggers --routines --event --set-gtid-purged=off > all_databases.sql

We can also select to dump a particular database we would like to restore:

mysqldump -u root -p --databases mysql > mysql.sql

After mysqldump completes we need to shut down MySQL and back up /var/db/mysql directory.

Now we need to switch off MySQL recovery mode in /usr/local/etc/mysql/my.cnf: 

innodb_force_recovery           = 0

After this we need the MySQL server to create a fresh database structure using following command:

mysqld_safe --skip-grant-tables --initialize &

Once the MySQL server is up we can restore our databases using following command:

mysql < all_databases.sql

Finally shut down the MySQL server and restart it normally.

Categories
Databases

MySQL Changes When Moving Web Server

Sometimes it happens that the web server IP address changes and previously working database connection for a PHP engine such as WordPress suddenly stops working. There need to be a couple of SQL adjustments done to fix the issue.

  1. Update the web server privileges to use the data base:
grant all privileges on your-database-name.* to 'your-database-user'@'new-web-server-ip-address';
flush privileges;

2. Update the web server host in the MySQL table:

update user where user='your-database-user'@'new-web-server-ip-address' identified by 'your-database-password';

Categories
*BSD

FreeBSD Release Branch: Current To Stable

During initial installation of Project Trident it was based on FreeBSD CURRENT branch. As of time of migration from Project Trident to GhostBSD, FreeBSD CURRENT version was 13.0. GhostBSD does have a repository based on FreeBSD-13.0-CURRENT. However all of the latest updates are coming to FreeBSD-12.1-STABLE branch. Here is a simple walkthrough of what I had to do in order to achieve migrationg to the latest branch of GhostBSD:

  • System ABI is sourced either from /etc/pkg.conf, /usr/local/etc/pkg.conf or from /bin/sh by default. Update ABI as follows in /etc/pkg.conf or /usr/local/etc/pkg.conf. The configuration setting is an architecture triple: ::

ABI = FreeBSD:12:amd64

  • Bootstrap package repository:

sudo pkg bootstrap -f

  • Update repository cache:

sudo pkg update -f

  • Install updates from the package repository:

sudo pkg upgrade -f

  • Ensure essential GhostBSD packages are installed:

sudo pkg install -g ghostbsd

  • Reboot the system

sudo reboot

Notes:

  • Because this is a major system update some packages need to be re-installed. For example, I needed to (re)install Xorg input drivers in order to fix my issue with an unresponsive keyboard and mouse.

sudo pkg install xorg-drivers

Categories
Databases

Recovery Of A Corrupted MySQL Database

In case a MySQL database is corrupted such as only its internal state is corrupted but data itself is intact it may still be possible to recover a database. The symptom of such corruption is that MySQL has failed to load database tables due to corrupted or missing indexes. The sample error logging when such corruption has happened looks as follows:

2020-03-08T19:52:25.634251Z 0 [ERROR] [FATAL] InnoDB: Table flags are 0 in the data dictionary but the flags in file /var/db/mysql/ibdata1 are 0x4800!
0x7af00e <handle_fatal_signal+0x2ae> at /usr/local/libexec/mysqld
2020-03-08 15:52:25 0x804418000  InnoDB: Assertion failure in thread 34431139840 in file ut0ut.cc line 918
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery

To achieve the database recovery in this case a set of utilities from the Undrop-for-innodb repository is used.

CAUTION Please ensure backing up a MySQL database directory before proceeding with the below steps

Steps to recover MySQL tables in ibd format

  • Copy ibdata1 from /var/db/mysql/ to a working directory for the recovery procedure and run the following command to parse ibdata1 file:
./stream_parser -f ibdata1
  • Parse SYS_TABLES from ibdata1 pages:
./c_parser -Uf pages-ibdata1/FIL_PAGE_INDEX/0000000000000001.page -p mysqldb.bak/ -t dictionary/SYS_TABLES.sql 2> SYS_TABLES .sql | sort -nk 4,5 > dumps/mysqldb_recovered/SYSTABLES
  • Parse SYS_INDEXES from ibdata1 pages:
./c_parser -Uf pages-ibdata1/FIL_PAGE_INDEX/0000000000000003.page -p mysqldb.bak/ -t dictionary/SYS_INDEXES.sql   2> SYS_INDEXES.sql | sort -nk 4,5 > dumps/mysqldb_recovered/SYSINDEX
  • Copy <table_name>.ibd from /var/db/mysql/<database_name> to a working directory for the recovery procedure and parse a MySQL table ibd file:
./stream_parser -f <table_name>.ibd
  • Parse MySQL table from its corresponding .ibd parsed page index given its create table schema. The page index number can be found by looking up PRIMARY column value in the parsed dumps/SYSTABLE and correlating it with a corresponding row in the parsed dumps/SYSINDEX. Then using that information run the following command:
./c_parser -6f ./pages-<table_name>.ibd/FIL_PAGE_INDEX/0000000000000222.page -t create_<table_name>.sql > dumps/<table_name>.tsv 2> load_cmd.sql
  • Finally load a recovered table into MySQL:
mysql --loose-local-infile=1 -u root -p -D <database_name> < load_cmd.sq

The steps are to be repeated for each corrupted table

Steps to recover MySQL tables in ISAM format

Usually these steps apply to a WordPress database

  • Go to the following directory /var/db/mysql/<database_name> and run the following command:
myisamchk <table_name>
  • If there is an error for a table from a previous command, then run the followinng command:
myisamchk -q -r  <table_name>
  • If .frm file is lost copy <table_name>.frm from a currently working MySQL database
Categories
Linux RAID

Recovering RAID-1 Array From The Missing Superblock Error

CAUTION Please ensure you have additional backup of your data prior to continuing the recovery

On Ubuntu Linux I have RAID-1 set up for various data backups. One day when I attempted to boot to Ubuntu I failed to do so. After some preliminary troubleshooting it was found that most of RAID-1 array members could not be mounted. When inspected further the mdadm tool, a tool used to create RAID-1 array, showed that superblocks for affected RAID array members were missing.

The following page describes in detail how to recover a RAID array in similar circumstances: Irreversible mdadm failure recovery The command that resolved this particular problem was:

mdadm --create /dev/md/vol<RAID array member #> --level=1 --chunk=512K --metadata=1.2  --data-offset=262144s   --raid-devices=2 /dev/sd<disk 1 of RAID-1> /dev/sd<disk 2 of RAID-1> --assume-clean --readonly

Notes:

  • As a result the RAID partitions have been mounted in read-only mode which is at least sufficient to ensure data is intact.
  • One suspicion about what could cause such the error is that after the latest update the Ubuntu fsck was erroneously run against RAID array. To ensure fsck is not run against RAID array the Pass column in the /etc/fstab configuration file needs to be set to 0 as follows.
# Device                Mountpoint              FStype          Options Dump Pass
UUID of disk      mountpoint            ext4             rw              0       0