• How to send a single UDP packet with Netcat

    At work today I needed this command to test UDP traffic to a particular endpoint, and thought I’d leave it written down here!

    This netcat command sends a single UDP packet with the string “foo” to a particular IPv4 endpoint on a particular UDP port:

    echo -n “foo” | netcat -u -w1 <host_ip> <port>
  • Using Ansible to provision LXC containers

    Since all of us have a bit of extra time lately I grabbed a copy of Jeff Geerling‘s Ansible for DevOps to finally tackle the project to translate my many, many bash scripts into something for flexible and idempotent.

    Ever since I began moving a lot of my cloud-hosted projects and applications to my own hardware in a colocation facility, I’ve spent a lot of time spinning up linux containers on Proxmox and running bash scripts to get them into a state that’s useful for me. This can get tedious and time consuming!

    My first project with Ansible was to not only streamline the creating of linux containers on my Proxmox hypervisor, but to provision them with the base configuration they need—and to subscribe them to my FreeIPA domain for central authentication.

    Besides the included README, you can find the entire playbook on GitHub.

    ansible-pve-lxc

    ansible-pve-lxc is an Ansible playbook and set of roles for provisioning and configuring LXC containers in a Proxmox virtual environment, and subsequently subscribing them to a FreeIPA domain.

    Define inventory with container configurations

    This inventory is in the YAML format, but will also work in INI format

    ---
    
    proxmox:
      hosts:
        pve.site.domain
    
    containers:
      hosts:
        db1.site.domain:
          ansible_host: 10.0.10.125
        app1.site.domain:
          ansible_host: 10.0.10.126

    Populate host variables with individual container configurations

    host_vars/db1.site.domain.yml

    ---
    
    node: 'gold'
    vmid: '110'
    cores: '1'
    memory: '1024'
    swap: '0'
    onboot: true
    network: '{"net0":"name=eth0,ip=10.0.10.125/24,gw=10.0.10.1,ip6=dhcp,bridge=vmbr1"}'

    host_vars/app1.site.domain.yml

    ---
    
    node: 'gold'
    vmid: '111'
    cores: '2'
    memory: '2048'
    swap: '0'
    onboot: true
    network: '{"net0":"name=eth0,ip=10.0.10.126/24,gw=10.0.10.1,ip6=dhcp,bridge=vmbr1"}'

    Configure variables

    Define PVE API and LXC default variables

    group_vars/proxmox.yml

    ---
    
    api_host: 'pve.site.domain'
    api_user: 'root@pam'
    
    defaults:
      node: 'gold'
      storage: 'local-zfs'
      cores: '1'
      memory: '1024'
      onboot: true
      pubkey: '{{ ssh_pubkey }}'
      searchdomain: 'site.domain'
      nameserver: '1.1.1.1,8.8.8.8'
      ostemplate: 'bulk:vztmpl/centos-8-20200402_amd64.tar.gz'
      network: '{"net0":"name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr1"}'

    Define PVE API password, SSH public key, and IPA admin password

    group_vars/all.yml

    ---
    
    api_password: '<proxmox_api_password>'
    ssh_pubkey: '<ssh_public_key>'
    ipa_domain: 'ipa.site.domain'
    ipa_realm: 'IPA.SITE.DOMAIN'
    ipa_principal: 'admin'
    ipa_admin_pw: '<ipa_admin_password>'

    Optional (but suggested!): Use ansible-vault to encrypt group_vars/all.yml

    $ ansible-vault encrypt group_vars/all.yml

    Use ansible-pve-lxc

    Optional: Temporarily disable ssh host key checking

    $ export ANSIBLE_HOST_KEY_CHECKING=False

    Run playbook

    $ ansible-playbook -i inventory site.yml --ask-become-pass

    Run playbook while asking for the vault password interactively

    $ ansible-playbook -i inventory site.yml --ask-become-pass --ask-vault-pass

    Optional: Re-enable ssh host key checking

    $ unset ANSIBLE_HOST_KEY_CHECKING
  • yeah - CLI tool for looking up OUIs

    A background-goal of mine this last month has been to learn enough of Rust to code something useful that I can use at work. In spare moments since the new year I’ve been reading “the book”—which has been more than helpful in learning the basics of Rust and what makes it such an interesting, powerful language.

    Fast-forward to today and I’m happy to share the useful something that I managed to make with Rust. Below is the README from yeah‘s GitHub page:

    yeah

    yeah is a command-line tool to return the vendor name for a given MAC address. Queries are ran against the IEEE OUI vendor list.

    Functionality

    Supports complete and partial MAC address queries

    Install yeah

    With crates.io:

    cargo install yeah

    From source:

    cargo install --path /path/to/yeah/repo

    Or, download and run from .zip or .tar.gz in releases

    Use yeah

    Print vendor of given MAC address

    $ yeah <mac>

    Options:

      -h, --help          Prints help information
      -v, --version       Prints version information
  • Enabling unsupported SFP+ transceivers on the Intel X520/Dell R720

    Having recently set up a new-to-me Dell R720 in a colocation provider’s environment—in my quest to “own” the whole stack from metal to service—I ran into an issue with not being able to use generic or Cisco-coded SFP+/SFP trancseivers in the included Intel X520 network card. dmesg kept indicating that the transceiver was unsupported. Because of impatience and other factors, I decided to dig a bit and see what it would take to enable support for good ‘ol grab bag generic SFP+/SFP transceivers.

    I came across this article on the ServeTheHome forums where user NathanA had already dug to the bottom of this issue, and fortunately came out the other side with a solution.

    NathanA discovered—via datasheets, mailing lists, and discussion threads—that a particular bit in a particular byte of the NIC EEPROM needs to be flipped in order to allow any SFP. In the Linux ixgbe driver, the byte in question is referred to as IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP.

    The steps to flip this bit, and thus allowing “unsupported” SFPs, is as follows:

    Note: I am in no way liable for any permanent damaged caused by running the following commands on your Intel NIC!

    Identify current value of offset 0x58

    raymond@server:~$ sudo ethtool -e eno1 offset 0x58 length 1
    Offset          Values
    ------          ------
    0x0058:         fc

    Write the new value to the card’s EEPROM

    The new hex value below is identical to simply flipping the last binary bit in the fc hex value.

    raymond@server:~$ sudo ethtool -E eno1 magic 0x10fb8086 0x58 value 0xfd

    Let’s check if we were successful

    If the output of the command below returns the new value of our 0x58 offset, we should be good to go!

    raymond@server:~$ sudo ethtool -e eno1 offset 0x58 length 1
    Offset          Values
    ------          ------
    0x0058:         fd

    What’s best about this EEPROM change is that is survives reboots, poweroffs, and will enable usage of unsupported SFPs in non-Unix operating systems.

    And, to return the EEPROM to its original state, simply write the old hex value back to the 0x58 offset and call it a day.

    1https://forums.servethehome.com/index.php?threads/patching-intel-x520-eeprom-to-unlock-all-sfp-transceivers.24634/
  • gpg-agent is older than us

    I use pass for password management and I ran into the following error when trying to retrieve a passphrase from its GPG store this morning:

    gpg: WARNING: server 'gpg-agent' is older than us (2.2.13 < 2.2.15)

    To fix this mismatch in version (after an update to GnuPG was installed) you need to kill the running version:

    gpgconf --kill -v gpg-agent

    After that pass should work once more, and not throw the “older than us” error.

    1https://medium.com/@doronsegal/gpg-agent-is-older-than-x-e8860a383cb0
  • Automating a Chinese robot vacuum

    A Roomba–or equivalent robot vacuum–is something I never felt I needed to purchase for myself. I have an old broom from colleg and a decent, budget upright vacuum. But, what happens when you’re gifted with a robot vacuum…?

    Behold, the holiday-gifted Ecovacs Deebot N79:

    A good boy

    Once rated Wirecutter’s Best Budget Robot Vacuum, the Deebot N79 fills me with both pride and guilt as it conquers the 2D world of my apartment. Whether it’s effortlessly chewing up cat litter, or getting irreparably tangled in lamp cords.

    Deebot has been a great addition to our home, but brings with it a few unavoidable facts:

    -The iOS app is garbage and provides no HomeKit support
    -The included handheld remote works, but speaks IR to Deebot (and can’t control him around corners)

    Trying to set a schedule on the handheld remote itself only works with near line-of-sight. And the schedule settings in the iOS app provide almost zero granularity. It provides no fine-tuned control of how long Deebot runs, whether it does full-room or edge cleaning, etc.

    Make Deebot do the WiFi

    In order to control Deebot over the network, we need to connect Deebot to the network. The instructions here speak for themselves.

    Once you’ve created your Ecovacs account on a server instance somewhere in China, try using the app to do the Deebot basics–cleaning, returning to the charge station, and directional control.

    Now Deebot is yet another WiFi device in your home.

    Install sucks

    William Pietri on GitHub created a set of python scripts that allow you to control your Ecovacs robot vacuum from the command-line.

    Install them with pip:

    pip install sucks

    Let’s log in to the Ecovacs servers with the credentials you just created:

    sucks login
    Ecovacs app email: [your email]
    Ecovacs app password: [your password]
    your two-letter country code: us
    your two-letter continent code: na
    Config saved.

    With that all set-up, let’s try to make Deebot do stuff from the CLI:

    sucks clean 5
    
    sucks clean 20 edge 15
    
    sucks stop
    
    sucks charge

    Cron your vacuum

    Lastly, let’s make a cron entry to run Deebot every day at 4PM local time:

    crontab -e
    0 16 * * * /<wherever_you_put_the_executable>/sucks clean 15 edge 10

    The above entry will–everyday at 16:00–command Deebot to perform a normal clean operation for 15 minutes, followed by an edge operation for 10 minutes, then instruct Deebot to return to its charging home.

    What next?

    Pietri’s project also includes a python library, so you can further add programatic logic to your vacuum.

    Maybe you could:

    -Have Deebot wake you up in the morning
    -Have Deebot keep your cats company
    -Have Deebot run a cleaning operation whenever Mom calls

  • Enable AirPods pairing on Arch Linux

    It’s beyond easy to pair AirPods with a non-Apple device. And on Arch, it’s trivial to have the AirPods appear in the bluetooth manager (in this case, blueman-applet)–but when you try to initiate pairing, it does not succeed.

    The trick here–if you don’t require pairing with Bluetooth LE devices–is to manually specify that the bluetooth controller (bluez) use BR/EDR transport rather than allow both transport modes.

    Open a shell and do the following:

    sudo vi /etc/bluetooth/main.conf

    Uncomment and manually set ControllerMode to BR/EDR:

    # Restricts all controllers to the specified transport. Default value
    # is "dual", i.e. both BR/EDR and LE enabled (when supported by the HW).
    # Possible values: "dual", "bredr", "le"
    ControllerMode = bredr
    

    Restart the bluetooth service:

    sudo systemctl restart bluetooth

    Try re-pairing!

  • Mac-like copy/paste on Fedora

    2018 is the year of Linux on the desktop—but only for me and only since I started working at a new company where I use Fedora for ~ eight hours daily. Coming from 10+ years of using Macs at work, the biggest change is using Ctrl + C/V for copy/paste and not Cmd + C/V. A couple of weeks in and my hand began to get a slight cramp from using my pinky and index fingers rather than my thumb and index fingers for the operation.

    After reading how to remap keys with xmodmap, the configuration below is what I came up with.

    Once parsed by Xorg, this lines in this file will remap right alt to right super, left alt to right control and left super to left alt.

    $ vi ~/.Xmodmap
    clear control
    clear mod1
    clear mod4
    keycode 133 = Alt_L Meta_L
    keycode 37 = Control_L
    keycode 64 = Control_R
    keycode 108 = Super_R
    add control = Control_L Control_R
    add mod1 = Alt_L Meta_L
    add mod4 = Super_R

    Confirm that this file is in your home directory, logout, log back in, and test. This may break existing shortcuts, so be sure to remap as neccessary under Settings > Keyboard > Application Shortcuts.

  • UPS Monitoring with NUT + Net-SNMP

    2020-05-22: After recently re-following these guidelines myself, a few changes have been made in how we start the UPS driver

    Early last year a friend and I started a local, wireless mesh network that eventually evolved into a community-run WISP. The full story of how it came to be is for another post…

    Fast forward many months and we secured some tower space for our radios and some limited rack space for our main networking equipment and servers. The location was just about perfect, but rather inconvenient to service.

    Secret internet lair
    Definition of not accessible

    Needing some reliable—but cheap—backup power for our main router, switch, and small 1U server we settled on the 10-outlet CyberPower 850VA/510W UPS (CP850PFCLCD). It provides reliable, stable output and will run our equipment (and radio) for about 45 minutes in the event the mountain site loses power.

    When I drove up to connect it to our equipment I plugged in the supplied USB Type-B to Type-A cable from the UPS to an empty port on our server—not yet knowing how to query the UPS, or if there was any linux tooling available for our particular device.

    Below are the steps I took to enable quantifiable monitoring of my consumer-grade UPS via open-source tooling on CentOS 7. And, in the footnotes I’ve linked the two pages that helped me succeed in this project.

    Install + configure NUT on CentOS 7

    Throughout this guide I’m using CentOS, but NUT should be available on most distros/package managers.

    Enable EPEL repository and install NUT

    $ sudo yum install epel-release
    $ sudo yum install nut

    Configure NUT driver

    If you’re not using the same UPS, check the NUT hardware compatibility list to see if your equipment is supported by the drivers included with NUT.

    First, run the command below to detect any supported UPS devices, and determine which driver is being used.

    $ sudo nut-scanner -U

    Output wil be similar to:

    [nutdev1]
        driver = "usbhid-ups"
        port = "auto"
        vendorid = "0764"
        productid = "0501"
        product = "CP850PFCLCD"
        serial = "000000000000"
        vendor = "CPS"
        bus = "001"

    Next, let’s configure NUT:

    $ sudo vi /etc/ups/ups.conf

    Append these lines:

    [cyberpower]
        driver = usbhid-ups
            port = auto
            desc = "CyberPower 850"

    Change [cyberpower] to any name you’d like. This value becomes the query name for polling the device. Additionally, driver should reference the output of nut-scanner and desc is entirely optional.

    Let’s start the UPS driver:

    $ sudo systemctl enable nut-driver.service
    $ sudo systemctl start nut-driver.service

    If you receive any error ensure you have sudo privileges—and try restarting the machine if the problem persists.

    We should also set NUT’s mode to standalone.

    $ sudo vi /etc/ups/nut.conf
    MODE=standalone

    From their documentation:

    standalone: This mode address a local only configuration, with 1 UPS protecting the local system. This implies to start the 3 NUT layer (driver, upsd and upsmon) and the matching configuration files. This mode can also address UPS redundancy.

    Configure NUT server and monitor

    Once the driver is up and running, we need to set up the user for accessing the NUT daemon.

    $ sudo vi /etc/ups/upsd.users

    Append:

    [local_mon]
    password = <secretpass>
    upsmon master

    Change <secretpass> to a secure password of your choosing.

    Lastly, we’ll edit the NUT monitor configuration with the name of our device, and the access information for the newly created user.

    $ sudo vi /etc/ups/upsmon.conf

    And append:

    MONITOR cyberpower@localhost 1 local_mon <secretpass> master

    Of course, change cyberpower to the name you gave your device, and <secretpass> to the password you specified in /etc/ups/upsd.users

    Enable services

    To ensure monitoring ability after planned and unplanned reboots, we need to enable and start the NUT system services.

    $ sudo systemctl enable nut-monitor.service
    $ sudo systemctl enable nut-server.service
    $ sudo systemctl start nut-server.service
    $ sudo systemctl start nut-monitor.service

    If you encounter any errors starting the services, restarting may resolve these. Often, running upsdrvctl start as we did earlier hijacks USB communication with the UPS and causes an error when trying to start NUT services.

    Test UPS querying!

    Run (replacing cyberpower with your device name):

    $ upsc cyberpower

    And, if all configurations are happy you will see some data returned:

    battery.charge: 100
    battery.charge.low: 10
    battery.charge.warning: 20
    battery.mfr.date: CPS
    battery.runtime: 2790
    battery.runtime.low: 300
    battery.type: PbAcid
    battery.voltage: 16.0
    battery.voltage.nominal: 24
    device.mfr: CPS
    device.model: CP850PFCLCD
    device.serial: 000000000000
    device.type: ups
    driver.name: usbhid-ups
    driver.parameter.pollfreq: 30
    driver.parameter.pollinterval: 2
    driver.parameter.port: auto
    driver.version: 2.7.2
    driver.version.data: CyberPower HID 0.3
    driver.version.internal: 0.38
    input.transfer.high: 139
    input.transfer.low: 88
    ...

    Install Net-SNMP

    $ sudo yum install net-snmp

    Create the script for MIB formatting of the query data:
    Note: this script formats MIBs specifically for LibreNMS.

    $ sudo vi /etc/snmp/ups-nut.sh

    Past the following, making sure to change UPS_NAME to your device name:

    #!/usr/bin/env bash
    
    UPS_NAME='cyberpower'
    
    PATH=$PATH:/usr/bin:/bin
    TMP=$(upsc $UPS_NAME 2>/dev/null)
    
    for value in "battery\.charge: [0-9.]+" "battery\.(runtime\.)?low: [0-9]+" "battery\.runtime: [0-9]+" "battery\.voltage: [0-9.]+" "battery\.voltage\.nominal: [0-9]+" "input\.voltage\.nominal: [0-9.]+" "input\.voltage: [0-9.]+" "ups\.load: [0-9.]+"
    do
        OUT=$(echo $TMP | grep -Eo "$value" | awk '{print $2}' | LANG=C sort | head -n 1)
        if [ -n "$OUT" ]; then
            echo $OUT
        else
            echo "Unknown"
        fi
    done

    Make the script executable:

    $ sudo chmod +x /etc/snmp/ups-nut.sh

    Now let’s extend this script to the SNMP daemon:

    $ sudo vi /etc/snmp/snmpd.conf

    Append the following and change <community> to your community name:

    rocommunity <community>
    extend ups-nut /etc/snmp/ups-nut.sh

    Let’s enable and start the SNMP daemon:

    $ sudo systemctl enable snmpd
    $ sudo systemctl start snmpd

    And finally, make a test query from a another host that can reach the server:

    $ snmpwalk -v2c -c <community> <host> 'NET-SNMP-EXTEND-MIB::nsExtendOutLine'

    You should see results similar to this:

    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".1 = STRING: 100
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".2 = STRING: 300
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".3 = STRING: 2790
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".4 = STRING: 16.0
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".5 = STRING: 24
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".6 = STRING: 120
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".7 = STRING: 125.0
    NET-SNMP-EXTEND-MIB::nsExtendOutLine."ups-nut".8 = STRING: 14

    Yeah!!

    Your host can now receive an SNMP query, poll your device with NUT, and respond with MIBs formatted for display/graphing on LibreNMS, Observium, or other SNMP-based monitoring platforms.

    You could also skip SNMP entirely and use a bash or python script to automate the retrieval of the UPS values via the upsc command.

    I hope this helps anyone looking to quantify and monitor the health of their UPS from afar.

    How I display my data

    I use both LibreNMS and Grafana (via LibreNMS + InfluxDB) to display information about my UPS.

    LibreNMS
    LibreNMS

    Grafana
    Grafana

    1https://www.luzem.com/2015/01/25/install-ups-in-centos-7/
    2https://community.librenms.org/t/ups-monitoring-with-nut/2313