1. Updating FreeBSD the Manual Way

    A few years ago I stopped updating my FreeBSD servers the standard way and instead follow the same method that I use in mkjail.sh

    freebsd-update is a fine tool, but it's long past its expiration date. The original purpose was to save bandwidth and only ship binary diffs to clients, but bandwidth is not a common issue anymore. The main issue is that freebsd-update requires making thousands of HTTP requests which can take a long time if the update server has high latency and we aren't quite ready for a full rollout of pkgbase yet. So here's the cheat code to fast updates.

    1. Download the base.txz, src.txz, and kernel.txz for your target RELEASE.

    2. Move /boot/kernel to /boot/kernel.old; you may have to clear out old kernels first.

    3. Extract the kernel

    tar -C / -xvpf kernel.txz
    

    4. Reboot into the new kernel.

    5. Extract base.txz, do not clobber /etc

    tar -C / --exclude=etc --clear-nochange-fflags -xvpf base.txz
    

    6. Extract src.txz. Clean out /usr/src first if you have to.

    tar -C / -xzvpf src.txz
    

    7. Run etcupdate

    cd /usr/src
    etcupdate
    

    8. Run your pkg upgrades

    pkg upgrade
    

    9. Clean out old libs and files that were removed since the update

    cd /usr/src
    yes | make delete-old
    yes | make delete-old-libs
    

    10. Reboot

    This can mostly be scripted, which I do. I can update a server completely in a couple minutes. I don't know of any Linux distro that can do a major upgrade this fast.

  2. Allowing Third Party Transceivers

    Here is how you enable use of third party or unsupported transceivers such as SFP / SFP+ fiber optics on e.g., Juniper's JunOS and Cisco IOS.

    Juniper:

    set chassis allow-other-transceivers
    

    Cisco:

    service unsupported-transceiver
    

    HP/Aruba:

    allow-unsupported-transceiver
    

    Dell:

    allow unsupported-transceiver
    

    Arista:

    Arista Networks EOS shell
    
    [admin@localhost ~]$ touch /mnt/flash/enable3px
    [admin@localhost ~]$ sudo reboot
    

    This is something seasoned network engineers know about, but there's no reason why this should be kept secret.

    I will add more when I learn of them.

    Last Updated 2024-01-04

  3. iPhone 11 Pro Has Broken Exif Orientation Data

    I have the new iPhone 11 Pro. It's a great camera. Turns out all of the photos I've taken so far have had incorrect EXIF Orientation data. This really sucks. I've confirmed the same issue happens on my wife's phone.

    It's shocking that this has not been noticed by Apple before the phone was released to the public. Have any photographers actually examined the images they shot on this phone?

    Here's an example image (JPEG format, for non-Apple folks and Safari won't even load HEIF images):

    Photo

    Here's the data dumped by exiftool:

    ExifTool Version Number         : 11.69
    File Name                       : IMG_0543.heic
    Directory                       : .
    File Size                       : 1728 kB
    File Modification Date/Time     : 2019:10:09 13:15:19-05:00
    File Access Date/Time           : 2019:10:11 13:36:55-05:00
    File Inode Change Date/Time     : 2019:10:11 13:36:03-05:00
    File Permissions                : rw-------
    File Type                       : HEIC
    File Type Extension             : heic
    MIME Type                       : image/heic
    Major Brand                     : High Efficiency Image Format HEVC still image (.HEIC)
    Minor Version                   : 0.0.0
    Compatible Brands               : mif1, miaf, MiHB, heic
    Handler Type                    : Picture
    Primary Item Reference          : 49
    Exif Byte Order                 : Big-endian (Motorola, MM)
    Make                            : Apple
    Camera Model Name               : iPhone 11 Pro
    Orientation                     : Rotate 90 CW 👈👈🚨🚨🚨🚨🚨
    X Resolution                    : 72
    Y Resolution                    : 72
    Resolution Unit                 : inches
    Software                        : 13.1.2
    Modify Date                     : 2019:10:02 19:19:06
    Y Cb Cr Positioning             : Centered
    Exposure Time                   : 1/15
    F Number                        : 1.8
    Exposure Program                : Program AE
    ISO                             : 1600
    Exif Version                    : 0231
    Date/Time Original              : 2019:10:02 19:19:06
    Create Date                     : 2019:10:02 19:19:06
    Offset Time                     : -05:00
    Offset Time Original            : -05:00
    Offset Time Digitized           : -05:00
    Components Configuration        : Y, Cb, Cr, -
    Shutter Speed Value             : 1/15
    Aperture Value                  : 1.8
    Brightness Value                : -3.833215521
    Exposure Compensation           : +0.0156
    Metering Mode                   : Multi-segment
    Flash                           : Off, Did not fire
    Focal Length                    : 4.2 mm
    Subject Area                    : 2002 1505 2213 1324
    Run Time Flags                  : Valid
    Run Time Value                  : 7694890327000
    Run Time Scale                  : 1000000000
    Run Time Epoch                  : 0
    Acceleration Vector             : 0.09105698762 -0.9248749617 -0.3692156373
    Sub Sec Time Original           : 313
    Sub Sec Time Digitized          : 313
    Flashpix Version                : 0100
    Color Space                     : Uncalibrated
    Exif Image Width                : 4032
    Exif Image Height               : 3024
    Sensing Method                  : One-chip color area
    Scene Type                      : Directly photographed
    Exposure Mode                   : Auto
    White Balance                   : Auto
    Focal Length In 35mm Format     : 26 mm
    Scene Capture Type              : Standard
    Lens Info                       : 1.539999962-6mm f/1.8-2.4
    Lens Make                       : Apple
    Lens Model                      : iPhone 11 Pro back triple camera 4.25mm f/1.8
    Profile CMM Type                : Apple Computer Inc.
    Profile Version                 : 4.0.0
    Profile Class                   : Display Device Profile
    Color Space Data                : RGB
    Profile Connection Space        : XYZ
    Profile Date Time               : 2017:07:07 13:22:32
    Profile File Signature          : acsp
    Primary Platform                : Apple Computer Inc.
    CMM Flags                       : Not Embedded, Independent
    Device Manufacturer             : Apple Computer Inc.
    Device Model                    :
    Device Attributes               : Reflective, Glossy, Positive, Color
    Rendering Intent                : Perceptual
    Connection Space Illuminant     : 0.9642 1 0.82491
    Profile Creator                 : Apple Computer Inc.
    Profile ID                      : ca1a9582257f104d389913d5d1ea1582
    Profile Description             : Display P3
    Profile Copyright               : Copyright Apple Inc., 2017
    Media White Point               : 0.95045 1 1.08905
    Red Matrix Column               : 0.51512 0.2412 -0.00105
    Green Matrix Column             : 0.29198 0.69225 0.04189
    Blue Matrix Column              : 0.1571 0.06657 0.78407
    Red Tone Reproduction Curve     : (Binary data 32 bytes, use -b option to extract)
    Chromatic Adaptation            : 1.04788 0.02292 -0.0502 0.02959 0.99048 -0.01706 -0.00923 0.01508 0.75168
    Blue Tone Reproduction Curve    : (Binary data 32 bytes, use -b option to extract)
    Green Tone Reproduction Curve   : (Binary data 32 bytes, use -b option to extract)
    HEVC Configuration Version      : 1
    General Profile Space           : Conforming
    General Tier Flag               : Main Tier
    General Profile IDC             : Main Still Picture Profile
    Gen Profile Compatibility Flags : Main Still Picture, Main 10, Main
    Constraint Indicator Flags      : 176 0 0 0 0 0
    General Level IDC               : 90 (level 3.0)
    Min Spatial Segmentation IDC    : 0
    Parallelism Type                : 0
    Chroma Format                   : 4:2:0
    Bit Depth Luma                  : 8
    Bit Depth Chroma                : 8
    Average Frame Rate              : 0
    Constant Frame Rate             : Unknown
    Num Temporal Layers             : 1
    Temporal ID Nested              : No
    Image Width                     : 4032
    Image Height                    : 3024
    Image Spatial Extent            : 4032x3024
    Rotation                        : 270
    Image Pixel Depth               : 8 8 8
    Movie Data Size                 : 1765488
    Movie Data Offset               : 4362
    Run Time Since Power Up         : 2:08:15
    Aperture                        : 1.8
    Image Size                      : 4032x3024
    Megapixels                      : 12.2
    Scale Factor To 35 mm Equivalent: 6.1
    Shutter Speed                   : 1/15
    Create Date                     : 2019:10:02 19:19:06.313-05:00
    Date/Time Original              : 2019:10:02 19:19:06.313-05:00
    Modify Date                     : 2019:10:02 19:19:06-05:00
    Circle Of Confusion             : 0.005 mm
    Field Of View                   : 69.4 deg
    Focal Length                    : 4.2 mm (35 mm equivalent: 26.0 mm)
    Hyperfocal Distance             : 2.04 m
    Light Value                     : 1.6
    

    Note, I have Location data turned off for my camera, so no GPS related stuff in here.

    And here's what ViewExif on iOS shows, which agrees:

    Photo

    Can someone please beg Apple to fix this? Completely unacceptable for this to let slip into production.

  4. Percona Toolkit Information Disclosure Vulnerability

    Percona includes an information disclosure vulnerability in the form of a "version check" feature in many of their products. Every time you run a database backup with xtrabackup or use any of the Percona Toolkit scripts the following information is collected and posted to https://v.percona.com:

    • OS Platform and Version
    • Perl version and version of modules
    • MySQL database version
    • Hostname of your server, obfuscated with md5_hex()
    • Presumably your IP address, visible in the logs

    You can find where it was added to xtrabackup with this commit. This same code is duplicated throughout the Percona Toolkit scripts.

    CVE-2014-2029 was the first CVE for this functionality which was regarding the ability for an attacker to MITM the connection and through injection achieve command execution. Later, CVE-2015-1027 was assigned as the fix of adding HTTPS was susceptible to a downgrade attack. At this time the "feature" that could allow command execution was already removed, but the information leakage was still present.

    Nobody seemed to care that Percona was collecting this data, however.

    I've contacted the Percona security team and requested that this feature be removed in its entirety. I'm already working on patches to rip it out of the ports/packages on FreeBSD, but you are vulnerable on other platforms. The only available workaround is to ensure that you pass --no-version-check to these utilities to disable this functionality, but most people will not see this blog post or be aware of the data collection being taken place.

    Originally the intention of this functionality was to inform the user of available software updates and discover if there are known vulnerabilities in your MySQL software, but that doesn't explain why this information is POSTed to their server. It's simply unacceptable.

  5. vBulletin cannot login without "Remember Me"

    If you happen to run a vBulletin forum and hit an issue where you cannot login to the site without first selecting the "Remember Me" checkbox, would you happen to be on CloudFlare or be using a reverse proxy? Make sure for CloudFlare you have the list of their upstream proxy IPs up to date or your REMOTE_ADDR of the client IP might not be getting set correctly.

    You can find the latest list of CloudFlare proxy IPs here:

    https://www.cloudflare.com/ips-v4

    https://www.cloudflare.com/ips-v6

    Do not trust the sample configs found on the CloudFlare site. They are almost always out of date.

  6. Git Is Not Revision Control

    Git has always rubbed me the wrong way. The ability to rewrite history and not tracking file renames are a few of the reasons it has turned me sour, not to mention the awful inconsistent UX as brilliantly mocked in Git Koans. I'm not objective enough to come up with a solid case against git as a revision control system which is why this FreeBSD developer email from phk resonated with me enough that I flagged it and re-read it many times over the last year.

    There was an internal discussion about the possible merits of switching to git to increase user contributions. This was his brilliant response, published with his permission:

    On Sun, Feb 26, 2017, at 14:59, Poul-Henning Kamp wrote:
    It is fundamentally wrong to ask "SVN or Git ?"
    
    SVN is obviously a Version Control System, it has all the classic
    attributes of one, including such crucial elements as progressing
    version number a definitive timeline and imutability.
    
    Git is clearly not a VCS, and it has never tried to be one, and
    people calling it one doesn't change that.
    
    The absense of a progressing version number and lack of a definitive
    timeline, not to mention all the many "unnatural acts" you can do
    to a git repo are sufficient arguments to settle this point.
    
    No, Git is something else, it is a colaboration tool.
    
    Git is a tool which allows people and projects to manage, modify,
    fork and merge the many different views, instances, variations and
    modifications of a work in progress across barriers of distrust.
    
    The crucial word there was "many different", which is the exact
    opposite of what a VCS strives for.
    
    A lot of the features Git provides, features which are what makes
    it great as a colaboration tool, flies in the face or or directly
    invalidates the guarantees you normally expect from a VCS, most
    notably progression of time & version, immutability and consistency
    of view.
    
    But in many cases Git is an adequate substitute for a VCS, you just
    have to augment it with an out-of-band definition of which tree is
    the 'definitive', and settle who gets to define what 'a version'
    means.  This is why github exists in the first place.
    
    In FreeBSD we have insisted on "proper version control" from
    day one, 23 years ago, and while it is a decision we should
    revisit periodically, everytime it has come up, it has been
    overwhelmingly confirmed as "the way we do things here".
    
    And this this thread, which is far from our first on the subject,
    fails to converge:  One side desires better colaboration tools and
    the other side is not willing to give up good old-fashioned version
    control to get it, both parties failing to realize that neither SVN
    nor Git will ever be able to do both, because the requirements are
    fundamentally different and in conflict with each other.
    
    So the task at hand, if there still is one, is to ask how we can
    make it easier to use Git as a colaboration tool for our committers
    and down-stream users.
    
    Poul-Henning
    
    PS: A good place to start would be to "bless" the github mirror
    and make sure the pull requests there get dealt with:
    
        https://github.com/freebsd/freebsd/pulls
    
    -- 
    Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
    phk@nospamplz           | TCP/IP since RFC 956
    FreeBSD committer       | BSD since 4.3-tahoe    
    Never attribute to malice what can adequately be explained by incompetence.
    

    I originally intended to cite specific points made in this email but honestly it's too good to not publish in its entirety. It's impossible to refute this line of reasoning in my opinion.

  7. FreeBSD Remote Serial Console Access With Dell and Cisco Servers

    I have become allergic to Java. It seems every time I need to access a server console my system is throwing fits about Java security. I've spent hours trying to fix a Java issue which was preventing me from fixing a server I needed console access to. I will show you how to end this madness.

    FreeBSD

    You should always enable the serial console on your servers. You never know when you will need it, and it is a prerequisite for this exercise. On FreeBSD you will want to do the following:

    /boot/loader.conf:

    boot_multicons="YES"
    boot_serial="YES"
    comconsole_speed="115200"
    console="comconsole,vidconsole"
    

    Enable ttyu0 always by changing onifconsole to on in /etc/ttys:

    ttyu0   "/usr/libexec/getty 3wire"      vt100   on secure
    

    On releases before 10.3-RELEASE you also change std.9600 to std.115200 as 3wire is not available:

    ttyu0   "/usr/libexec/getty std.115200"      vt100   on secure
    

    When complete you will need to enable it with kill -HUP 1. You can find more details on this in the FreeBSD handbook.

    Dell DRAC

    On Dell servers you will need to enable the serial console in the BIOS. If you have multiple serial ports the following configuration should allow both the physical and virtual serial ports to function as intended. Here is a screenshot of my BIOS settings:

    Photo

    You will also want to enable IPMI. This can be done in the DRAC settings on boot in the following screenshot, or you can SSH to the DRAC console and enable it there with the rest of the settings.

    Photo

    To finish the setup we SSH to the DRAC console and run the following commands:

    # The DRAC shell is usually prefixed with something like /admin1-> but
    # I will not print it here so you can easily copy/paste this
    racadm config -g cfgSerial -o cfgSerialBaudRate 115200
    racadm config -g cfgSerial -o cfgSerialCom2RedirEnable 1
    racadm config -g cfgSerial -o cfgSerialSshEnable 1
    racadm config -g cfgIpmiSol -o cfgIpmiSolEnable 1
    racadm config -g cfgIpmiSol -o cfgIpmiSolBaudRate 115200
    racadm config -g cfgIpmiLan -o cfgIpmiLanEnable 1
    

    There are settings to limit DRAC network access (HTTPS, SSH, IPMI) to a specific subnet, but I strongly suggest you do not place your DRAC on the internet and protect it with a real firewall. It would be reasonable to enable this bruteforce protection if you wish to do so. The following blocks for 5 minutes after 5 failed attempts:

    racadm config -g cfgRacTuning -o cfgRacTuneIpBlkEnable 1
    racadm config -g cfgRacTuning -o cfgRacTuneIpBlkFailCount 5
    racadm config -g cfgRacTuning -o cfgRacTuneIpBlkFailWindow 60
    racadm config -g cfgRacTuning -o cfgRacTuneIpBlkPenaltyTime 300
    

    Now you have remote access to your server without Java. You can access it with one of the following:

    ssh:

    $ ssh root@1.2.3.4
    /admin1-> console com2
    

    ipmi:

    $ ipmitool -I lanplus -U root -H 1.2.3.4 sol activate
    

    Both of these will render best if your terminal window is 80x24.

    Cisco CIMC

    Assuming you've already provisioned an IP address for your Cisco CIMC and it is accessible on the network you can simply ssh into the CIMC and run the following commands:

    cisco-cimc# scope sol
    cisco-cimc /sol # set baud-rate 115200
    cisco-cimc /sol *# set enabled yes
    cisco-cimc /sol *# commit
    show
    cisco-cimc /sol # show
    Enabled Baud Rate(bps)  
    ------- --------------- 
    yes     115200          
    cisco-cimc# top
    cisco-cimc# scope ipmi
    cisco-cimc /ipmi # set enabled yes
    cisco-cimc /ipmi *# commit
    cisco-cimc /ipmi # show
    Enabled Encryption Key                           Privilege Level Limit 
    ------- ---------------------------------------- --------------------- 
    yes     0000000000000000000000000000000000000000 admin      
    db02-ipmi /ipmi # 
    

    Or for your copy/paste speedrun:

    scope sol
    set baud-rate 115200
    set enabled yes
    top
    scope ipmi
    set enabled yes
    commit
    

    Now you can connect to the serial console by one of the following:

    ssh:

    $ ssh admin@1.2.3.4
    cisco-cimc# connect host
    

    ipmi:

    $ ipmitool -I lanplus -U admin -H 1.2.3.4 sol activate
    

    The Cisco CIMC also allows you to mount remote media from a fileshare or HTTP/HTTPS URL which is fantastic for troubleshooting. :-)

  8. Using FreeBSD as a Time Capsule for OSX

    I've had both a coworker and a FreeBSD developer ask me recently how to use FreeBSD as a Time Capsule for Time Machine from OSX. There are a lot of tutorials out there and most of them are non-functional. This is possibly the simplest guide that is known to be working. It uses local unix account authentication for the shares over AFP, and of course it's backed by ZFS.

    pkg install netatalk3 avahi-app
    

    /usr/local/etc/afp.conf:

    [Global]
    vol preset = default_for_all_vol
    log file = /var/log/netatalk.log
    hosts allow = 172.16.1.0/24 2001:470:1f11:1e8::/64
    mimic model = TimeCapsule6,116
    
    [default_for_all_vol]
    #file perm = 0640
    #directory perm = 0750
    cnid scheme = dbd
    #ea = none|auto|sys
    ea = ad
    
    [backup-mark]
    path = /local/timecapsule/mark
    valid users = feld
    time machine = yes
    

    /usr/local/etc/avahi/services/afp.service:

    <?xml version="1.0" standalone='no'?><!--*-nxml-*-->
    <!DOCTYPE service-group SYSTEM "avahi-service.dtd">
    <service-group>
    <name replace-wildcards="yes">%h</name>
    <service>
    <type>_afpovertcp._tcp</type>
    <port>548</port>
    </service>
    </service-group>
    

    /etc/rc.conf:

    # time machine
    dbus_enable="YES"
    netatalk_enable="YES"
    afpd_enable="YES"
    cnid_metad_enable="YES"
    avahi_daemon_enable="YES"
    

    I'm aware Apple is moving away from AFP to SMB, but I haven't investigated what it would take to make this work with Samba instead of Netatalk.

  9. Generating DDNS TSIG Keys for BIND

    The tutorials on how to generate TSIG keys for BIND DDNS updates suck. It would also be tedious if tasked to generate several. I'm not sure why ISC has not produced a standalone script or utility to make this easier as nobody should have to piece it together by hand.

    I was attempting to explain to a coworker how to generate his key and when I couldn't find an easier way I decided to just write something myself. So here you go, a terrible perl script to produce HMAC-SHA256 TSIG keys. Go hog wild.

    #!/usr/local/bin/perl -w
    # This script is overkill, but at least it's easier than
    # explaining to people how to use ddns-keygen
    
    use warnings;
    use strict;
    use Digest::SHA qw(hmac_sha256_base64);
    use Bytes::Random::Secure qw(random_bytes);
    
    # As these are one-offs and we don't need a reusable secret key, we make
    # both the key and the data random. 512 bytes of entropy ought to be
    # enough for everybody...
    my $data = random_bytes(512);
    my $key  = random_bytes(512);
    
    my $digest = hmac_sha256_base64( $data, $key );
    
    # Fix padding of Base64 digests
    while ( length($digest) % 4 ) {
        $digest .= '=';
    }
    
    print qq[
    key "changeme" {
            algorithm hmac-sha256;
            secret "$digest";
    };
    ];
    

    With slightly more effort you could make the TSIG key format configurable as well as allow you to provide a key name as flags.

  10. Speeding up MySQL Import on FreeBSD

    I was recently tasked with rebuilding a readonly slave database server which only slaves a couple of the available databases. The backup/dump is straightforward and fast, but the restore was being excruciatingly slow. I didn't want to wait a week for this thing to finish, so I had to compile a list of optimizations that would speed up the process. This is the best way to do it on FreeBSD, assuming you're working with InnoDB. Additional optimizations may be required if you're using a different database engine.

    Please note this is assuming no other databases are running on this MySQL instance. Some of these are rather dangerous and you wouldn't want to put other live data at risk.

    my.cnf:

    innodb_buffer_pool_size = 38G  # roughly 70-80% of your available memory
    innodb_flush_method = O_DIRECT
    sync_binlog = 0 # Don't keep this permanently
    innodb_flush_log_at_trx_commit = 0 # Don't keep this permanently
    innodb_log_file_size = 1G
    innodb_log_buffer_size = 256M
    innodb_write_io_threads = 16
    

    /etc/rc.conf:

    # Only use this during imports
    mysql_args="--innodb-doublewrite=0"
    

    The actual import command in use:

    # cat dump.sql.gz | { echo "set sql_log_bin=0; set autocommit=0; set
    unique_checks=0; set foreign_key_checks=0;"; zcat; } | mysql -u root -p
    

    And now I've gone from a tens of MBs imported per minute to several GBs imported per minute.

Page 1 / 3