Wimpy's World

faffing about with diffs

martin@wimpress.com (Martin Wimpress) — Thu, 05 Jun 2025 13:37:42 +0100

Picture this: you’ve just spent literal minutes crafting the perfect bit of code, you’re chuffed with yourself, and then you open your git client only to stare at that commit message box. 🤔

That cursor blinking mockingly while you try to remember what you changed just moments ago. And you wonder, why do I have to write this? git already knows what happened!

Now, I’m a proper GitKraken devotee – have been for years 🦑 Love the thing so much I’ve disabled all git integrations in VS Code to maintain a clean separation between coding and source control. It’s brilliant software and I’m happy to pay for it (you should too using my handy-dandy referral link) 🤑

When AI promises go sideways

GitKraken recently added AI integration for generating commit messages, complete with customisable prompts. Naturally, I thought “Right, this’ll sort my commit message faff once and for all!” Two problems emerged rather quickly:

First, I blew through my AI credit bundle faster than a student with their first overdraft facility. Turns out I commit quite a lot, and those credits don’t stretch as far as you’d hope.
Second, and this really wound me up, I could not for the life of me get it to generate proper Conventional Commits. I know how to craft AI system prompts – I’ve been mucking about with LLMs long enough – but GitKraken’s AI seemed determined to ignore my carefully crafted instructions about feat:, fix:, and chore: prefixes.

Enter faff, stage left

So naturally, being the sort of person who builds entire projects out of mild irritation, I decided to solve this myself. Meet faff – yet another bloody AI commit generator, because apparently the Internet wasn’t drowning in enough of them already ☔

faff does one thing: it looks at your staged git diff, sends it to a local Ollama LLM, and spits out a conventional commit message. No cloud APIs, no credit bundles, no sending your TODO: delete this abomination code comments to feed the vibe-coding dealers.

git add .
faff

Perfect conventional commits, every time. Well, most of the time. It’s AI, not magic 🪄

The plot twist that made me question everything

Here’s where this story takes a turn that’ll make you laugh at my expense. After building faff, documenting it, and feeling rather pleased with myself, I discovered something that made me want to bang my head against my desk repeatedly.

You can hook GitKraken’s AI up to Ollama.

And naturally, my carefully crafted system prompt from faff, when used in GitKraken AI via Ollama with the same qwen2.5-coder models, works exactly like faff does. 🤦‍♂️

So technically, I could have solved my original problem without building an new tool. But where’s the fun in that?

Why faff still matters (and it’s not just stubbornness)

Despite this rather embarrassing revelation, faff still has its place:

Privacy by default: Everything stays local, until you push it to GitHub
No vendor lock-in: Works with any Ollama-supported model. Want to try the latest coding model? Just change the environment variable.
Shell-first design: Perfect for those of us who live in terminals, or an excuse for me creating another shonky shell script
Learning project: Born from genuine frustration and a desire to poke around the Ollama API. Sometimes the best tools come from scratching your own itch.

Plus, let’s be honest – there’s something deeply satisfying about having a tool that does exactly what you want, how you want it, without any faff! 🥁

Give it a whirl

If you’re tired of crafting commit messages that either say “Updated stuff” (again!) or read like Victorian novels, faff might be worth a look. It’s a shell script, it works with local AI, and it follows conventional commits standards without the existential crisis.

And if you decide you’d rather stick with GitKraken’s AI (which, after my discoveries, is perfectly reasonable), at least grab it through my referral link so I can pretend this whole adventure was planned from the start. ️🗺️

Because sometimes the best projects are the ones born from pure spite, accidentally becoming useful along the way. That’s half the fun of building things, isn’t it?

Drop the faff, dodge the judgment, get back to coding. 🦙

Nothing but Nix

martin@wimpress.com (Martin Wimpress) — Thu, 15 May 2025 13:37:42 +0100

Have you ever tried to build a complex NixOS ️❄️ configuration in GitHub Actions only to be greeted by the dreaded “no space left on device” error? I certainly have, and it’s been driving me bonkers for quite some time! 😖

Standard GitHub Actions runners come with a paltry ~20GB of free disk space, or so it would seem. Which sounds decent until you try to build sophisticated NixOS or Home Manager configurations. A comprehensive workstation or home lab server setup can consume 10-15GB due to all the included packages and dependencies, leaving precious little headroom for anything else 🤏

This space limitation effectively meant I couldn’t build my full workstation and server configurations in CI. So annoying! 😠 I wanted my CI to cache complete builds to FlakeHub Cache, but was forever stuck with only partial builds or individual packages that wouldn’t properly test the complete configurations. This meant that each configuration change on my workstations and servers required frustrating additional compilation time for things like 3rd party kernel modules, Ollama, custom package overrides, and more - all because I couldn’t get full builds cached through CI ️🥺

Well, I’d had enough of that nonsense and decided to solve the problem once and for all 🧠

Introducing Nothing but Nix ️❄️

I’m chuffed to bits to announce Nothing but Nix, a GitHub Action that brutally reclaims disk space 🪓 on runners and transforms them into Nix powerhouses!

Here’s what it does in a nutshell:

- uses: actions/checkout@v4
- uses: wimpysworld/nothing-but-nix@main
- uses: DeterminateSystems/determinate-nix-action@main
- run: |
  nix flake check

The result? Instead of ~20GB for your Nix store, you get 65GB to 130GB of sweet, glorious Nix-dedicated space. That’s enough to build even the chonkiest of configurations! 🪨

Space, the Final Frontier 🌌

Working at Determinate Systems, I get to see firsthand how powerful Nix can be when properly cached. Our FlakeHub Cache makes system updates lightning fast ⚡ but I couldn’t fully leverage it in CI because my builds kept running out of space.

I wanted to be able to build and test all my configurations in CI:

💁 Servers: media services, local LLMs, website, fediverse services, distributed storage, backup
️🖥️ Workstations: dual GPU, loaded with dev tools, content creation apps and local LLMs
💻 Laptops: optimized for travel, media on the go, presentations and development
👻 VMs: for Linux desktop development and testing

But GitHub’s ~20GB limitation kept getting in the way 🙅 Sure, I could have paid GitHub for their larger runners with more disk space, but being both cheap and stubborn, I wasn’t having any of that! 💸❌ Instead, I started thinking, “Hang on a minute, those free runners must have more space somewhere - I just need to find it and claim it!” 🤔💭

After digging through the runner specs and file system layout, I discovered that GitHub Actions runners have a large chunk of free space on the /mnt filesystem that’s barely used 🤯 There was also the possibility to reclaim significant space by purging pre-installed software that Nix users don’t need anyway 😈

The Slightly Mad Science Bit ‍🧑‍🔬

At its core, Nothing but Nix uses a two-pronged attack:

1. The Initial Slash: Instant Volume Creation

First, the action creates a large loop device from free space on /mnt and sets up a properly tuned BTRFS filesystem:

# Create a large disk image in the free space
free_space=$(df -m --output=avail /mnt | tail -n 1 | tr -d ' ')
loop_dev=$(sudo losetup --find)
sudo fallocate -l $((free_space - 1024))M "/mnt/disk0.img"
sudo losetup "${loop_dev}" "/mnt/disk0.img"

# Set up an optimized BTRFS filesystem
sudo mkfs.btrfs -L nix -d raid0 -m raid0 --nodiscard "${loop_dev}"
sudo mount LABEL=nix /nix -o noatime,nobarrier,nodiscard,compress=zstd:1,space_cache=v2,commit=120

This immediately provides around 65GB of Nix-ready space before your workflow even gets going! 💪 I could have stopped here, but did I mention beging stubborn yet? 🫏

2. The Background Purge: Merciless Space Reclamation

While your workflow continues, Nothing but Nix starts a background process to ruthlessly eliminate unnecessary software:

Docker images? Gone! 🗑️
Language runtimes? Obliterated! 💥
Package managers? Annihilated! 💣
Documentation? Vaporized! 🔥

As space is reclaimed, it creates a second disk image on the root file system and adds it to the BTRFS pool, dynamically growing your Nix volume up to around 130GB. 🚀

The file system purge is powered by rmz (from the Fast Unix Commands (FUC) project) - a high-performance alternative to rm that makes deletion blazingly fast. Using traditional rm a full file system purge took ~11 minutes ⏳😞 rmz cut that to under 60 seconds! ️⏱️😀

BTRFS with `nodiscard`?

I’m using BTRFS for the volume because it:

Supports dynamic device addition
Has built-in compression to save even more space
Allows optimal space utilization with RAID0 layout

The nodiscard mount option is absolutely essential because we’re using loop devices backed by sparse files.

Without it, BTRFS would try to issue TRIM/discard commands that cause allocation size to be misreported when using loop devices. This misreporting would result in inaccurate space accounting. The nodiscard option ensures the filesystem maintains an accurate picture of its available storage.

Choose Your Weapon: The Hatchet Protocol 🪓

Not everyone needs the same level of space reclamation, so I’ve added different “Hatchet Protocol” levels to control how aggressive the action is:

- uses: wimpysworld/nothing-but-nix@main
 with:
 hatchet-protocol: 'cleave' # Options: holster, carve, cleave (default), rampage

Protocol	`/nix`	Description
Holster	~65GB	Keep the hatchet sheathed, use space from `/mnt`
Carve	~85GB	Craft and combine free space from `/` and `/mnt`
Cleave	~115GB	Make decisive cuts to large packages
Rampage	~130GB	Relentless elimination of all bloat

I recommend Cleave for most users, which is the default setting. It strikes a good balance between reclaiming space and keeping some useful tools around. But if you’re a proper Nix enthusiast who believes that #nix-is-life, the Rampage protocol will squeeze out every last byte of space for your Nix store ️️❄️

From Frustration to Freedom ️🕊️

Since implementing Nothing but Nix, I’ve been able to build all my NixOS and Home Manager configurations in CI. This means:

🔒 Every flake.lock update is fully tested against my complete configuration set
✅ All successful builds are cached to FlakeHub Cache
⚡ System updates are lightning-fast

This has been an absolute game-changer for my workflow. Now when I apply updates to any of my devices, everything is delivered directly from cache.flakehub.com with zero local compilation time! 🚀 All packages have been pre-built and verified in CI, meaning updates that would have taken ages to compile (looking at you ollama 👀) locally now complete in seconds. It’s the difference between waiting for a coffee break and a blink-and-you’ll-miss-it experience 🤩

Here’s what the build times look like:

💁 2x Servers: ~5-10 minutes
️🖥️ 2x Workstations: ~6-13 minutes
💻 4x Laptops: ~7-10 minutes
👻 2x Virutal Manachines: ~4-10 minutes
📦 59x Local packages: ~5-14 minutes

Getting Started with Nothing but Nix ✨

Ready to give it a go? Here’s how to use it in your GitHub Actions workflow:

name: "Test Nix Flake"
on:
 pull_request:
 push:
 branches: [ main ]

jobs:
 tests:
 runs-on: ubuntu-latest
 permissions:
 id-token: write
 contents: read
 steps:
 - uses: actions/checkout@v4
 # 👇 Add this before installing Nix
 - uses: wimpysworld/nothing-but-nix@main
 - uses: DeterminateSystems/determinate-nix-action@main
 - run: |
 # Now you have 100+ GB for your Nix builds!
 nix build . -L
 nix flake check

For FlakeHub users, make sure you add the permissions block as shown above. This allows Determinate Nix to authenticate with FlakeHub and FlakeHub Cache.

Beyond Storage to Virtual Machines? 🔮

Now that I’ve proven large volumes can be dynamically constructed in GitHub runners, I’m starting to think about what else might be possible. A GitHub action that implements something like Quickemu (another project of mine), to standup KVM-accelerated VMs inside a GitHub runner is my favourite idea 💡

GitHub has recently announced hardware acceleration for Android virtualization on Linux runners, which means the KVM infrastructure is already there and avilable for open source projects and larger runners 😯 All we need to do is leverage it for our own VMs.

Imagine being able to run a full VM of your validated build environment inside GitHub Actions! It would be like having a self-hosted runner, but without the self-hosting 😏 And it just so happens that the Determinate Nix Action enables KVM in GitHub runners. Coincidence? 😉

Build Something Massive ️🏗️

Nothing but Nix removes one of the most frustrating limitations for Nix users in GitHub Actions. No more trimming your configurations to fit into tiny spaces - now you can build and test your full NixOS fleet with confidence 🤓

Give it a try, and let me know what you think!

The action is available at:

GitHub: wimpysworld/nothing-but-nix
Marketplace: Nothing but Nix

If you run into any issues or have ideas for improvements, please open an issue on GitHub. And if this tool saves your CI builds, consider giving the repo a star! ⭐

Happy Nixing! ❄️

Fuzzel Colour Picker

martin@wimpress.com (Martin Wimpress) — Tue, 15 Apr 2025 13:37:42 +0100

We’ve all been there - you’re knee-deep in customizing your desktop, tweaking configs, perfecting that theme, and suddenly you need to grab a specific colour from somewhere on screen. If you’re like me and recently made the jump to Hyprland (or other contemporary Wayland compositor), you might have found yourself missing some of those handy X11 utilities that “just worked” ️™️

The Wayland Colour Picker Problem

When I switched my daily driver to Hyprland back in August 2024, I was chuffed with the smooth animations, efficient tiling, and general polish. But as with any big change, there were a few splinters in my workflow that needed sorting.

One particular rough edge was the lack of a proper GUI colour picker. On X11, I’d been spoiled by my mate Stuart Langridge’s excellent tool Pick - a proper colour picker that did exactly what it said on the tin. But in Wayland? Well, I was left running hyprpicker from the terminal like some kind of savage! 😅

hyprpicker in the terminal

Don’t get me wrong - hyprpicker works brilliantly for what it is. But after seeing how well my recent integrations of bzmenu and iwmenu worked with Fuzzel, a lightbulb moment hit me: why not create a proper Fuzzel-based front-end for hyprpicker? 💡

Birth of fuzzel-hyprpicker

After my second coffee one morning (the optimal programming fuel ☕💪), I decided to tackle this problem head-on. If I was going to spend hours staring at my Catppuccin Mocha themed desktop, I needed proper tools to maintain it! ‍🐈‍⬛🎨

The goal was simple:

Create a Fuzzel-based UI for hyprpicker
Maintain persistent colour history
Make it keyboard-friendly
Automatcially copy selected colours to clipboard

The result is fuzzel-hyprpicker.sh - a little script that brings together the power of hyprpicker with the elegant UI of Fuzzel.

fuzzel-hyprpicker showing the colour selection menu

How It Works

The magic behind fuzzel-hyprpicker.sh is surprisingly straightforward. Let me walk you through the key bits:

Setting Up the Basics

#!/usr/bin/env bash
# A tool to pick colours from the screen using hyprpicker and fuzzel

set +u # Disable nounset
APP_NAME="fuzzel-hyprpicker"
NOTIFY="notify-desktop --app-name=$APP_NAME --icon=org.gnome.design.Palette"

# Set up the storage directory and file
CONFIG_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/$APP_NAME"
HISTORY_FILE="$CONFIG_DIR/colours.txt"
ICONS_DIR="$CONFIG_DIR/icons"

Nothing fancy here - we’re just setting up our environment, defining where we’ll store our colour history, and setting up notifications. Note that I properly follow the XDG spec here - it’s the little things! 😉

The Clever Bit: SVG Icons

Here’s where things get interesting. Fuzzel supports icons in its menus, but I needed a way to show colour swatches. The solution? Dynamically generated SVG files:

function generate_svg_icon() {
 local colour="$1"
 local icon_path="$ICONS_DIR/$colour.svg"

 # Create an SVG for the colour if it doesn't exist
 if [ ! -f "$icon_path" ]; then
 cat > "$icon_path" <<EOF
<svg width="128" height="128" xmlns="http://www.w3.org/2000/svg">
 <rect width="128" height="128" fill="#$colour" />
</svg>
EOF
 fi
}

I’m rather pleased with myself for this solution. Each time you pick a colour, it generates a tiny SVG file that’s just a coloured square. Fuzzel then displays this next to the hex code in the menu. Well done brain! 🧠

The menu itself is built dynamically, showing an eyedropper icon for picking a new colour followed by your colour history:

function build_menu() {
 echo -e "Pick a colour\0icon\x1f$ICONS_DIR/eyedropper.svg"
 # Add history items if they exist
 if [ -s "$HISTORY_FILE" ]; then
 while read -r colour; do
 # If the preview icon doesn't exist, generate it
 if [ ! -e "$ICONS_DIR/$colour.svg" ]; then
 generate_svg_icon "$colour"
 fi
 # Display the colour with a preview
 echo -e "#$colour\0icon\x1f$ICONS_DIR/$colour.svg"
 done < "$HISTORY_FILE"
 fi
}

This uses the rofi “protocol” for menu items, something Fuzzel has built-in support for 💖 The \0icon\x1f bit is how you tell Fuzzel “hey, use this icon for this menu item”.

Putting It All Together

The full workflow goes like this:

Press Ctrl + Alt + K to open the picker
Select “Pick a colour” or choose one from your history
If picking a new colour, hyprpicker does it’s thing and click anywhere on screen
The colour gets automatically copied to your clipboard
A notification appears showing the selected colour

hyprpicker colour selection

Integrating It With Your Setup

Want to give this a try yourself? Here’s how to adapt it to your wlroots-compatible desktop:

First, make sure you have the following installed:
- hyprpicker
- fuzzel
- wl-clipboard (for clipboard management)
- notify-desktop (for notifications)
  - Although I’m planning on switching to fyi soon…
Save the script somewhere in your path (I’ve got mine as part of my Nix configuration) ️❄️
Make it executable:

chmod +x fuzzel-hyprpicker.sh

Bind it to a keyboard shortcut in your Hyprland config:

bind = CTRL ALT, K, exec, fuzzel-hyprpicker.sh

Give it a whirl!

It should work with any wlroots-compatible compositor, not just Hyprland.

Possible Enhancements

This is very much a v1 and there’s plenty of room for improvement:

Add support for different colour formats (RGB, HSL, etc.)
Implement colour name lookup
Create a proper project rather than just a script wedged in my Nix configuration

But for a morning hack, it’s been surprisingly useful! Reaching for Ctrl + Alt + K brings a smile to my face 😀

The Joy of Keyboard-Driven Workflows

What I love about tools like this is how they contribute to a cohesive, keyboard-driven workflow. Between Hyprland’s tiling, Waybar’s status info, and Fuzzel-based menus for everything from launching apps to picking colours, my fingers rarely stray from the keyboard these days.

There’s something deeply satisfying about crafting your own tools to scratch your own itches. It’s a long held Linux desktop tradition, isn’t it? Taking what works, improving what doesn’t, and sharing the results with others.

My hyprland desktop with fuzzel-hyprpicker in action

More Fuzzel-Based Tools?

I’ve created several other Fuzzel menus that integrate with my Hyprland desktop. I should probably post about those as well - emoji picker and screenshot and screen recording being among my most useful. Let me know in the comments if you’d be interested in seeing those! 👇

In the meantime, give fuzzel-hyprpicker.sh a try and let me know what you think. And if you improve upon it, please do share your changes!

The full script is available in my Nix config repository: fuzzel-hyprpicker.sh

Happy colour picking! 🎨🤏

Have you created any custom tools for your Wayland desktop? Are there any X11 utilities you miss since switching? Let me know in the comments below!

Install ZeroTier on Steam Deck

martin@wimpress.com (Martin Wimpress) — Thu, 17 Aug 2023 12:15:45 +0100

It took 9 episodes of Linux Matters for my friends and co-hosts Alan and Mark to wear down 😅 my resolve to buy a Steam Deck, even after I passed on my original Steam Deck pre-order. I’m glad they did, it’s a great little device and a dream come true for Linux 🐧 nerds; excellent hardware and completely hackable 🧑‍💻

I ordered the 64GB model via the recently announced Steam Deck™ Certified Refurbished program. I’m going to resist retrofitting a larger NVMe SSD, and have instead added a 1TB microSDXC card 💾 The 64GB internal eMMC storage will be used for SteamOS and the microSDXC card will be used exclusively for games 🕹️

While I’m avoiding hardware modifications, I do want to install some software enhancements on the Steam Deck; specifically ZeroTier. I spoke enthusiastically about ZeroTier in Linux Matters: Picking Apart the Steam Deck (Episode 8); it is as essential as WiFi for me. So, give that episode a listen to hear why I love ZeroTier so much 💖

Linux Matters Podcast

What follows is a guide to installing ZeroTier on the Steam Deck, but this technique can be used to install any software on Steam Deck that is available for Arch Linux 👍️

Installing ZeroTier on Steam Deck

The Steam Deck has a (mostly) read-only filesystem. We’ll use Rwfus: Read-Write OverlayFS for your Steam Deck to install ZeroTier so that it persists across reboots and SteamOS updates.

The author describes rwfus as:

a vinyl couch cover for your filesystem, Rwfus covers your Deck’s /usr/ directory (and some others) allowing you to initialize and use pacman (the Arch Linux package manager) on the Steam Deck without losing packages when the next update comes out.

Perfect 👌️

Enter the Desktop

The process of installing and configuring ZeroTier requires the use of the Steam Deck desktop mode. Connecting a mouse 🖱️ and keyboard ⌨️ to the Steam Deck will make this process much easier. After the Steam Deck has booted you should Switch to Desktop.

Press the “Steam” button on the Steam Deck and scroll down through the menu and select “Power”.
Then, select the “Switch to Desktop” option.
Once the desktop has loaded, click on the Steam Deck logo in the lower-left corner of the desktop to bring up the menu.
Select “System” in the left column and then “Konsole” from the right column.
- You can also press the Super key and type in “Konsole” in the search bar.
Suddenly, as if by magic, the terminal appeared 🪄

Konsole on Steam Deck

Set a Password

By default, no password 🔓️ is set for the deck user. With no password set, it is not possible to run commands in the root context using sudo. Type passwd in the terminal to set a password.

(deck@steamdeck ~) $ passwd
New Password:
Retype New Password:

Install Rwfus

With that done we can install rwfus.

git clone https://github.com/ValShaped/rwfus.git
cd rwfus
./rwfus -iI

Install ZeroTier

Run sudo pacman -Sy to update your repositories 🌿 Now install ZeroTier.

sudo pacman -S zerotier-one

Enable and Start ZeroTier service

ZeroTier installs its systemd unit in /usr/lib/systemd/system/, but as we’re installing while Rwfus is active that service file is installed into the Rwfus overlay. This is a problem because running systemctl enable zerotier-one now would create a systemd unit in /etc/systemd/system/multi-user.target.wants/ that references a file in /usr, and that file will not exist until Rwfus is running. The effect is that the ZeroTier service will not start on boot as systemd will not be able to find the service file.

But there is a simple workaround, copy the ZeroTier service file to /etc/systemd/system, which is fine as /etc is already a writable overlay.

cp /usr/lib/systemd/system/zerotier-one.service /etc/systemd/system/zerotier-one.service

Now, enable and start the ZeroTier service.

sudo systemctl enable zerotier-one --now

Join a ZeroTier Network

Find your ZeroTier device ID (a 10-digit hexadecimal number) by running:

sudo zerotier-cli info

The output will look 👀 something like this:

200 info deadbeef00 1.10.1 OFFLINE

Join a ZeroTier network using a network ID (a 16-digit hexadecimal number) which you can get from the My ZeroTier page.

sudo zerotier-cli join <network id>

Return to the My ZeroTier page select the network you joined, scroll down to the Members section and authorize the Steam Deck device ID.

Verify that the Steam Deck is correctly allocated an IP address, this may take up to 30 seconds ⏳️

ip --brief addr | grep ^zt

You should see 👁️ output similar to this:

zta1b2c3d4 UNKNOWN 192.168.192.59/24 baad::4a4a:faff:fefe:5aa5/64

Use your ZeroTier network

You can now connect from your Steam Deck to other devices on the same ZeroTier network. Or you could enable SSH on the Steam Deck and then log in to it from another device on the ZeroTier network 💪

sudo systemctl enable sshd --now

Conclusion

With ZeroTier installed on the Steam Deck, it is part of my network where ever I may roam; and enables some interesting possibilities such as remote play from anywhere in the world or playing LAN multi-player games with remote friends. Share in the comments how you’re using ZeroTier on your Steam Deck 💬

Steam Deck

I’ll be keeping my Steam Deck software modifications to a minimum 🤏 but where I do make changes I’ll be careful not to disrupt the SteamOS root file systems or potentially break SteamOS updates. I’m glad solutions like Rwfus exist to allow me to install the software I need. But…. 🤔

Nix is what I work on as my day job at Determinate Systems ❄️ I’m creating a little project that any Steam Deck owner can use to bring the power of Nix to Steam Deck while maintaining isolation from the SteamOS root file system. That project depends on the Determinate Nix Installer, which natively supports Steam Deck. So, if you want to get a head start with Nix on the Steam Deck then you can do that today 🙂

Steam Box vs Steam Deck

martin@wimpress.com (Martin Wimpress) — Mon, 01 May 2023 18:38:19 +0100

When I received my Steam Deck Purchase Information email I wasn’t playing games all that often; mostly online racing games with friends once a week. I didn’t think I’d get much use out of a handheld console. I also had an inventory of PC parts available that could be used to build a dedicated Steam Box. So on July 18th 2022, I declined spending £569 on a Steam Deck and set about building a Steam Box instead. Here’s how I got on and what I learned.

As featured on Linux Matters! 🎙️

I recently discussed why I cancelled my Steam Deck pre-order on the Linux Matters podcast. You can hear that discussion with my friends Alan and Mark in Linux Matters: Lawful evil monitor alignment (Episode 2).

Linux Matters Podcast

Hardware 🖥️

At the time I had an inventory of parts that weren’t being used and on the second desk in my office I had an ultra-wide monitor, keyboard, mouse and more controllers than I know what to do with. From the parts I had available I selected the following:

I had all the parts I needed to build a Steam Box, except for the case and power supply, so I bought a Fractal Design Node 202 (without PSU) for £65 on eBay and a CORSAIR SF750 80 PLUS Platinum SFX Power Supply for £145 on Amazon.

Node 202 Considerations 💼

The Node 202 is a compact chassis (10.2 litres) that can house a capable gaming build, but has several specific considerations to keep in mind.

GPU Considerations

The case is designed for 2-slot graphics cards, which means that larger triple-slot cards may not fit. The maximum supported dimensions are:

Length: 310mm
Width: 145mm
Depth: 47mm

The Radeon™ RX 5700 XT GAMING OC 8G I have is too big to fit in the Node 202 case, but I was able to remove the shroud and fans, which allowed me to fit the GPU in the case with 2x 120mm Noctua case fans mounted on the bottom panel of the case to provide cooling.

Deshrouded Gigabyte RX 5700XT Gaming OC

CPU Considerations

The Node 202 riser card interface is PCI Express 3.0 x16 and there is limited space for a CPU cooler. This is why I went with the 5700G APU I had on hand, instead of the 5900X that I also had in my supplies.

The 5700G APU is a 65W TDP part, requiring less cooling than the 5900X
- The Noctua NH-L9a-AM4 I had is a low-profile cooler that is rated for 65W TDP.
- I replaced the NH-L9a-AM4 fan with a Noctua NF-B9 redux-1600 PWM 4-Pin, 1600 RPM to provide additional cooling
The 5700G APU is a single CCX, further reducing the cooling requirements
The 5700G APU only has PCI Express 3.0, the same as the Riser card interface that comes with the Node 202 case
- A CPU with PCI Express 4.0 support or higher would work too, but would be limited to PCI Express 3.0 speeds

The Node 202 also has limited space for cable management. The modular design of the Corsair SF750 power supply along with its easy-to-route cables helped to keep the cables tidy and prevented them from blocking airflow.

Partially Built Steam Box

Operating System 📀

Initially, I installed HoloISO, which gets lots of attention but switched to ChimeraOS on a recommendation from my friend Jorge Casto. ChimeraOS is excellent! 🧑‍🍳🤌

“ChimeraOS is an operating system that provides an out of the box couch gaming experience. After installation, boot directly into Steam Big Picture and start playing your favorite games.

If you want Steam in your living room, you want ChimeraOS” - ChimeraOS Team

If you’re building a Steam Box, I highly recommend ChimeraOS. One of the key advantages of ChimeraOS is its use of atomic updates. Atomic updates ensure that updates are applied as a single, cohesive unit, which reduces the likelihood of update failures and ensures that the system is always in a consistent state. In contrast, HoloISO uses the regular Arch Linux update mechanism, which applies updates individually and can potentially lead to update failures. In pursuit of a true console-like experience, ChimeraOS gets it right and even supports rolling back updates should anything go wrong.

ChimeraOS is a very well-thought-out implementation of a console-like operating system when compared to HoloISO. Here’s a summary of how ChimeraOS, SteamOS and HoloISO compare:

	ChimeraOS	SteamOS 3.x	HoloISO
Based on	Arch Linux	Arch Linux	SteamOS 3.x/Arch Linux
Desktop	Gnome	KDE Plasma	KDE Plasma
Atomic updates	✔️	✔️	❌
Read-write root filesystem	❌	❌	✔️
Custom partitioning	❌	❌	✔️
General hardware support	✔️	❌	✔️
Up-to-date base packages	✔️	❌	❌
Remote app installation	✔️	❌	❌
Built-in EGS & GOG support	✔️	❌	❌
Built-in emulation support	✔️	❌	❌
Additional game tweaks	✔️	❌	❌
Additional artwork	✔️	❌	❌
Built-in GE Proton	✔️	❌	❌
Full source publicly hosted	✔️	❌	❌
Forkable infrastructure	✔️	❌	❌

Table taken from the ChimeraOS Wiki

ChimeraOS Experience 🦁

I’ve been using ChimeraOS for nearly a year and I’m extremely happy with it. I installed ChimeraOS 33 when I built the Steam Box and it has updated itself, without issue, to ChimeraOS 41 (at the time of writing). New releases are made every 4 to 6 weeks. Some of the ChimeraOS features that I love are:

A true Steam Deck-like experience but for a wide range of hardware
- More so if you have Steam Controllers available, I’ve got three connected wirelessly
Wake up from sleep via the push of a controller button
Built-in support for Epic Game Store and GOG
Built-in support for emulation
- I enjoy playing retro arcade and console games, and ChimeraOS makes it easy to install and play them
Built-in support for Xbox Wireless Controllers
- My Stadia Controllers with Bluetooth Firmware also work great with ChimeraOS
Bundles current builds of Proton-GE
Web-based UI for managing the system/games

Having a dedicated “games console” has resulted in me playing more games, but more importantly, playing more games with my family. The three of us sit together to play through games, often with my daughter on controls and my wife and I problem-solving and making suggestions. As a family, we’ve revisited Spyro™ Reignited Trilogy something my wife and I played through together on the original PlayStation last-century! 😱 My daughter and I are most of the way through the Monkey Island Collection and we’ve discovered we love co-op games like Unravel Two and It Takes Two. We’ve also purchased some “mobile” games we play on tablets and NVIDIA SHIELD such as Riptide GP2, Riptide GP Renegade and Horizon Chase Turbo. Yes, we like racing games 🏁

My daughter has access to (mostly) the same games her friends play on Xbox and PlayStation. She doesn’t feel like our Steam Box is “different” from her friend’s consoles. When they visit each other they broadly play the same selection of titles, except for platform exclusives of course. When Stray and Five Nights at Freddys became the games to play among her circle of friends, they were a click away and worked perfectly. Thanks to the Steam Deck, ChimeraOS is just a games console and not a “PC” to her. She doesn’t have to worry about updates, drivers, or anything else. She just turns it on and plays games.

ChimeraOS powered Steam Box in action - before I figured out how to enable ultra-wide resolution

I’ve found ChimeraOS offers much better game compatibility than I’ve been able to achieve by simply installing Steam on Ubuntu. For example, Forza Horizon 4 and Forza Horizon 5 both run beautifully on the ChimeraOS powered Steam Box; something that eluded me on Ubuntu. The Steam Box is connected to a 3440x1440 ultra-wide monitor and configuring each game to use “Native” resolution via the game properties unlocks a gorgeous ultra-wide gaming experience. There is no going back from this once you’ve experienced it, especially for split screen local multiplayer games.

Steam Deck 🎮

Several of my friends have Steam Decks and love them. They use them as both a handheld console and docked under the TV for family gaming. So, I could be using a Steam Deck to facilitate our family gaming requirements, but I don’t think I would have arrived at the idea of using a Steam Deck in this way without first experiencing the benefits of the Steam Box. I am considering getting a Steam Deck or similar handheld gaming-orientated PC and it would certainly be nice to use something more power efficient than the 270W total system power consumption the Steam Box requires. You can hear more about how my friend Mark has switched to Steam Deck as his primary gaming device in Linux Matters: Mastodon on My Résumé (Episode 1).

Upgrades 📈

Since I first built the Steam Box I’ve upgraded the GPU to a Fighter AMD Radeon™ RX 6700 XT 12GB GDDR6. At the time of writing, this is the most powerful Radeon GPU that comfortably fits in the Node 202 case and still leaves room for the additional case fans I installed. This GPU is a great match for the ultra-wide monitor and all the games I’ve tried can run at 3440x1440 with high presets (or better) at upwards of 60fps. When I first built the Steam Box, ChimeraOS worked best with Radeon GPUs, if you wanted to enable the Steam Deck UI (gamepadui). Support for Intel and NVIDIA GPUs is improving, but I’ve stayed with AMD Radeon because that’s been proven to be the most compatible option.

I’ve also added a second SSD; the Crucial P3 4TB M.2 PCIe Gen3. I used the ChimerOS web UI to prepare the drive and it now automatically shows up in the Steam UI as a Steam Library location.

What’s next? 🔮

The Steam Box has a permanent home in my office, but originally I’d planned to put it in the lounge under the TV. I have an original Steam Link, so now I’m thinking that I’ll put the Steam Link under the TV to replace the NVIDIA SHIELD. I have some older GPD devices that are designed for gaming, so I think I’ll put ChimeraOS on those to see how they perform as dedicated handheld game consoles. Could they be used as Steam Link clients too?

ChimeraOS also supports installing software via Flathub. I’d like to get a browser installed that can be used for streaming services like Netflix and Disney+ so we can use the Steam Box for watching TV too.

I’ll report back on all of this in a future post. Like 👍️ and Subscribe ❤️

Conclusion 🏁

The tinkerer, maker and developer in me wants to try NixOS as games console solution but I’m starting to think that other than the learning opportunity there is no real value in doing so, considering what an excellent job ChimeraOS does.

And here we are at the conclusion with barely a mention of Linux 🐧 ChimeraOS and the Steam Deck are excellent examples of Linux devices done right; all the Linux-y stuff is just implementation detail. Even I don’t think of our Steam Box as being a Linux-y thing; I just pick up a controller, wake up the console and start playing contemporary games or my favourite retro classics 🕹️ without any fuss.

Flash Stadia Controller Bluetooth Firmware on Linux

martin@wimpress.com (Martin Wimpress) — Fri, 28 Apr 2023 16:00:00 +0100

I was a Google Stadia “founder” and when it was shut down on January 18, 2023 I was left with 3 perfectly usable Stadia Controllers; provided I could connect them using a USB-C cable. I quite like the Stadia Controller, the form factor is similar to the PlayStation DualShock but with an Xbox style button layout.

Stadia Controller

While using it via USB-C is great for PC, wireless connectivity would provide more options. Thankfully Google released Bluetooth firmware for the Stadia Controller which disables Stadia’s proprietary Wi-Fi connectivity and enables Bluetooth® Low Energy (BLE). You can then use it like a standard wireless Bluetooth controller with other hardware and also still plug your Stadia Controller into a device with a USB-C cable.

Compatibility 🧩

The Stadia Controller Device Compatibility List says

“Testing of Bluetooth mode was focused on the devices and systems listed below, but may not work with all hardware configurations.”

Windows 10 and 11 + Steam
MacOS® 13 + Steam
ChromeOS
Android

Most importantly, it says “computers must have a Bluetooth Low Energy adapter in order to communicate with the Stadia Controller in Bluetooth mode and that hardware compatibility will vary.”

Flash Stadia Controller Bluetooth Firmware using Ubuntu 22.04 🐧

Here’s how I flashed that firmware on all 3 of my controllers using Linux; well Ubuntu 22.04 to be precise. The system requirements for the Bluetooth Firmware tool is stated to be Chrome 108 or newer. No problem! But, I was unable to get the controllers to show up in Chrome until some udev rules were added. Open a terminal and run the following command from any directory:

{ cat <<EOF
# SDP protocol
KERNEL=="hidraw*", ATTRS{idVendor}=="1fc9", MODE="0666"
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="1fc9", MODE="0666"
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="0d28", MODE="0666"
# Flashloader
KERNEL=="hidraw*", ATTRS{idVendor}=="15a2", MODE="0666"
# Controller
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="18d1", MODE="0666"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="18d1", ATTRS{idProduct}=="9400", MODE="0660", TAG+="uaccess"
EOF
} | sudo tee /etc/udev/rules.d/70-stadiacontroller-flash.rules

Now run the following to reload the udev rules.

sudo udevadm control --reload-rules
sudo udevadm trigger

With that done, I restarted Google Chrome and followed the instructions on the Stadia Controller Firmware Flashing Tool page. I was able to flash the firmware on all 3 of my controllers. The process is pretty straightforward but I’ve included the instructions below for reference.

Charge your controller

Make sure that your controller has been charged for 30 minutes and that you’re using a USB data cable.

Charge your controller

Unplug your controller to power it off

If it turns on again, hold the Stadia button for 4 seconds to power it off

Unplug your controller to power it off

Hold the option button (three dots) while plugging in your controller

The status light should remain off. If it turns on, unplug the controller and try again

Hold the option button (three dots) while plugging in your controller

Press the Option, Assistant, A, and Y buttons at the same time

There won’t be any controller feedback, proceed to the next step to confirm the controller is unlocked

Press the Option, Assistant, A, and Y buttons at the same time

Select your controller in the Chrome device list and click “Connect”.

The text in the flashing utility says the Stadia Controller should be named “SP Blank RT Family” or “Stadia Controller” or “USB COMPOSITE DEVICE”.

Select your controller in the Chrome device list again and click "Connect".

Download complete

You’ve almost finished. Go to the final step to install the latest version of Bluetooth mode on your controller.

Download complete

Install Bluetooth mode

One more time - select your controller in the Chrome device list, then click “Connect”. Installation will start automatically and should only take a minute.

The text in the flashing utility says the Stadia Controller should be named “USB COMPOSITE DEVICE” or “Stadia Controller” or “SP Blank RT Family”.

Install Bluetooth mode

Installation complete

Wait a minute or so for the firmware installation to complete.

Installation complete

Pairing Stadia Controller with Bluetooth

Hold the Y and Stadia buttons for 2 seconds until the status light flashes orange. It is now in pairing mode and visible to other devices.

Pairing Stadia Controller with Bluetooth

Hurray up! ⏳️

After the shutdown in January 2023, you have until December 31, 2023, to update the controller so it can use Bluetooth mode.

Fortunately, the Stadia Controller firmware has been archived and an Unofficial Stadia Controller Flashing utility created which might prove useful after December 31, 2023.

Unofficial Stadia Controller Flashing utility

Your mileage may vary using the re-implementation of the Utility for flashing Stadia Controllers, so if you’re doing this before December 31, 2023, use the official tool.

Conclusion

After updating my Stadia Controllers with Bluetooth firmware, I’ve successfully paired them with my Android tablet, iPhone 13 and ChimeraOS; an operating system that provides an out-of-the-box couch gaming experience 🎮️ While Stadia might be dead and gone, I’m very happy to continue to get utility out of the controllers; which turned out to be free after the reimbursement Google offered when Stadia was shut down.

Why I chose the ThinkPad Z13 Gen1 as my Linux laptop

martin@wimpress.com (Martin Wimpress) — Tue, 18 Apr 2023 14:30:26 +0000

During the second half of 2022, I dusted off my laptop and travelled to three events for the first time in over two years. During these trips, it became apparent that my laptop is not the right tool for the job.

At KubeCon EU 2022 my colleague Lindsay brought her Apple Macbook Pro M1. It was lightweight, compact, looked fabulous and had epic battery life. Meanwhile, my ThinkPad P1 Gen 1 looked fabulous but it is a bit of a chonker and a massive power pig 🔌🐖 Battery anxiety was constant that week and also on my subsequent trips to SREday 2022 and the Ubuntu Summit 2022. Sensing that 2022 wasn’t an outlier and more travel would be on the cards in 2023 I decided that I wanted some of that thin and light laptop action. In early December 2022, I went hunting for a Linux laptop and this is my journey.

As featured on Linux Matters! 🎙️

I recently discussed my hunt for a new Linux Loving Laptop on the Linux Matters podcast. You can hear that discussion with my friends Alan and Mark in Linux Matters: Mastodon on My Résumé (Episode 1).

Linux Matters Podcast

Past laptop purchasing mistakes 😱

“Those that fail to learn from history are doomed to repeat it” - Winston Churchill.

The requirements for my last two laptop purchases were very different from what I need today; a Dell XPS 15 5550 and ThinkPad P1 Gen1 Both have significant power requirements with 15.6" UHD displays and discrete NVIDIA GPUs. The ThinkPad P1 Gen also sports a Xeon CPU ⚡️ These made sense when I bought them, I was travelling one week every month and regularly compiling large applications, building container images, VMs and operating system images. These days I have a Threadripper 3970X workstation at home that I can connect to via Tailscale to run compute-intensive tasks. I simply don’t need a workhorse 🐴 laptop anymore.

Laptop criteria 📑

I’m deeply impressed with the outstanding work the Asahi Linux team are doing to enable Linux on Apple Silicon Macs, but running Linux on an M1 Mac isn’t viable for me as some hardware support (HDMI for example) is still a work in progress at the time of writing. Not ideal when you’re a conference speaker and running booth demos.

These are my criteria for the new laptop. Some must-haves, some nice-to-haves and some hard exclusions.

Fully Linux compatible.
- Linux pre-installed to demonstrate Linux is fully supported
Full working day battery life; ~8 hours in my opinion.
Low-power CPU, 35W or under
- Ideally AMD 6000 series but a 12th Gen Intel as a compromise
- No 11th Gen Intel or AMD 5000 series CPUs will be considered
64GB RAM, will compromise on 32GB RAM
13" or 14" 1920x1200 matte display
- No UHD resolutions display will be considered (for power-saving reasons)
- Touch support is a nice-to-have, but not essential
- 1920x1080 as a compromise, but nothing lower
No discrete GPU. Again for power-saving reasons.
USB-C charging
Dual NVME SSD, or at least a single 2TB (or more) SSD
Decent keyboard and touchpad
The Laptop should weigh close to 1kg
Premium build quality and design (somewhat subjective I know)

With this list of requirements established, I started collating Linux laptop comparison notes in this somewhat idiosyncratic spreadsheet

A spreadsheet that probably only makes sense to me

Looking at the list of laptops in the sheet above, you might be wondering why I didn’t consider any laptops from the established dedicated Linux laptop vendors such as Entroware, Slimbook and StarLabs. Well, I did look at everything they offered at the time and none of them had a model available that met the requirements I’ve outlined above or the estimated dispatch time was nearly half a year.

I’m not going to elaborate on the rationale behind ruling certain laptop models in or out, but I will say this; I was very impressed to see every laptop in the Lenovo ThinkPad lineup had a Linux pre-install option of either Ubuntu or Fedora in “Build YourPC” system configurator. While comparing the power requirements of Intel’s i7-12xx series and AMD’s 68x0 series CPU at the time, I was sold on the impressive battery endurance of AMD’s offerings and the superior integrated graphics, so I excluded any laptop with Intel CPUs quite early on.

Lenovo ThinkPad Z13 Gen 1 with Ubuntu pre-installed

I went with the ThinkPad Z13 Gen 1 with Ubuntu pre-installed, and the ThinkPad T14s Gen 3 was runner-up in my selection process.

Specifications 📝

The key specifications for the laptop I ordered are AMD Ryzen 7 PRO 6850U CPU, 32 GB LPDDR5-6400MHz (Soldered), 1 TB SSD M.2 2242 PCIe Gen4, 13.3" WUXGA (1920 x 1200), IPS, Anti-Glare, Non-Touch display weighing in at 1.19kg. I think this configuration hits the sweet spot for battery endurance, more on that later.

I’m not a fan of the increasing trend of soldering RAM on motherboards, but that was common across all the models of laptops I was considering. If Framework had offered an AMD 6000 series CPU option at the time, I would’ve had a Framework laptop on my short list as the modular design of the Framework laptops is very appealing.

And yes, I did make a compromise with the laptop specifications; that 1TB M.2 SSD is below my minimum requirement of 2TB. I did do my homework though and will present my creative upgrade solution in a future blog post. Like and Subscribe 😉

Build Quality & Design 💻️

Without a doubt, the ThinkPad Z13 Gen 1 is a gorgeous laptop. I do not have enough superlatives to express just how much I love it. It is, in my option, an almost flawless design. Exactly the compact form factor laptop I was seeking; it’s beautiful to look at from any angle and a delight to use. Here are some highlights.

ThinkPad Z13 Gen 1

The laptop is engineered from 75% recycled aluminium and 95% recycled plastics, then boxed in 100% renewable, compostable packaging. The touchpad is sublime and amazingly 120mm wide on the compact laptop. The haptic touch is simply excellent. The touchpad is the most Macbook-like touchpad I’ve used on any PC and it is so good it has caused me to change what desktop environment I now use. More on this in a future blog post. However, there is currently no Linux software to configure the haptic touchpad settings such as click force and touchpad feedback intensity but I’ve been perfectly happy with the defaults. If you do dual boot Windows the haptic settings configured via Windows are stored on the device and carry over to Linux.

The Keyboard is an excellent low-profile design, each key has ~1.2mm of travel which is the same as the actuation point on my Razer Huntsman V2 TKL keyboards with opto-mechanical switches. Most importantly the Fn key is in the correct place on the Z13 and not where ThinkPads have been incorrectly plonking it for years. Fight me! 🥊 If you are a long-time ThinkPad owner, it is likely you’ll hate the idea of the keyboard and touchpad on the Z13 since it is quite a departure from the traditional design. But I went into this with my eyes 👀 open and have watched and read many reviews.

"This is the way"

The Display is bright (400nits), anti-reflective and anti-smudge, covers 100% of the sRGB colour gamut and the laptop is perfectly balanced so it can be opened one-handed. I’m embracing the compact, thin and light lifestyle; so the two USB-C ports are fine with me as I’ve chosen a laptop configuration with excellent battery endurance (more on that in a bit) and plan to use it completely untethered most of the time with cables and adapters only plugged in for very specific tasks and overnight recharging.

I do have one niggle though; there is an unusable 2242 M.2 slot on the motherboard. It is only intended for use with select models of WWAN cards, none of which are a configuration option for the UK models of the Z13. It doesn’t look like the antenna is wired in either, so even if you do get a supported WWAN card aftermarket it is unlikely to work well; if at all. I can share my iPhone’s mobile service via Wifi, so not a deal breaker in that regard but it is rather annoying to have an M.2 slot on the motherboard and nothing I can do with it.

Ubuntu pre-install experience 👌

It is worth noting that at the time of purchase, selecting Ubuntu or Fedora in the system configuration on the Lenovo website applied a £155 discount! 🤑 Choosing Ubuntu or Fedora across the ThinkPad line applies a discount, although the amount varies based on the model.

All OEMs that partner with Ubuntu get an OEM designation for each device officially endorsed and supported by Canonical. The ThinkPad Z13 Gen 1 is known as Sutton Newell Abe and the oem-sutton.newell-abe-meta package (along with the associated OEM PPA) provides the device-specific hardware enablement and power management tuning.

ThinkPad Z13 Gen 1 is known as Sutton Newell Abe for OEM enablement

The OEM image of Ubuntu differs from the standard Ubuntu image in a few ways. The most obvious is that OEM image comes with recovery media creation pre-installed. This is a great feature and I highly recommend you use it to create a recovery USB stick, either during the initial setup or post-setup. The OEM image also comes with Chromium and Firefox installed, as opposed to just Firefox in the downloadable release of Ubuntu. IIRC, this is because Chromium has traditionally worked better with touchscreen laptops. The fingerprint reader works. I was able to enrol fingerprints quickly and authenticate GDM logins. But the biometric support does not extend throughout the system; Ubuntu Software doesn’t integrate with the fingerprint reader, nor does snapd or any other privilege escalation.

I also experienced the touchpad becoming unresponsive, requiring a click to “unfreeze” it. This happens quite frequently and detracts from what is otherwise an excellent, class-leading, touchpad. I suspect it is overly aggressive power management settings in the OEM image, but I didn’t investigate beyond that hunch. Both these issues with the fingerprint reader and touchpad are Ubuntu-specific and nothing related to the actual hardware. Both issues are absent when running NixOS.

Having a tier-1 vendor such as Lenovo ship a laptop with Ubuntu pre-installed has the benefit of great firmware support. I’ve received several firmware updates since I got the laptop, and they have all applied without issue.

Firmware updates for the ThinkPad Z13 Gen 1 on Ubuntu 20.04

Why not Ubuntu 22.04? 🤔

My Z13 was shipped with Ubuntu 20.04.4 with OEM optimised Linux kernel 5.14.0-1054-oem. Some might be perplexed (or annoyed) that it didn’t come with Ubuntu 22.04, given it was ordered in late 2022. I used to work for Canonical, and during that time worked with Lenovo to enable Ubuntu on ~60 of their laptops and workstations between 2019 and 2021, so I do have some insight into how this process works. It simply boils down to how the factory image certification process works, and once an image is qualified that is what ships on the device for the duration of its availability. Re-certification adds cost and takes time, so it’s extremely rare for a device to have a revised factory image qualified once it has gone on sale.

Upgrading to Ubuntu 22.04 LTS worked flawlessly, full audio support was restored and no other hardware supported regressed. Brilliant.

Linux Kernel 6.1 🌰

The pre-installed Ubuntu and Fedora images for the Z13 come with kernels that have backported patches applied to fully support the device. If you buy a ThinkPad Z13 and plan to run another Linux distro on it, make sure you can install Linux kernel version 6.1 or newer. This is to ensure the Qualcomm Wi-Fi 6E NFA725A 2x2 AX chipset is fully supported and that the required patches to properly suspend and resume are available.

Battery endurance 🔋

This is, after all, my most important selection criteria. The conclusion here is simple: 11 hours ⏱️

This is 11 hours of mixed-use. Coding in Visual Studio Code. Chatting in Slack and Discord. Video calls in Google Meet and Zoom. Compiling software. Some screen capture with OBS Studio and basic video editing with Shotcut. Closing the lid of the laptop and leaving it suspended for 24 hours depletes the battery by ~4% which is about 2 Wh. Very respectable. I haven’t felt the need to profile power consumption or tweak anything as I’m get plenty of untethered compute time. I’m very happy with the battery life, typically charging the laptop overnight while I sleep and running it all day battery only.

What’s next? 🔮

As noted, I did compromise on the 1TB M.2 SSD. I’ve come up with an aftermarket solution to upgrade to 2TB which I will post about soon. I chose a laptop pre-installed with Linux for two reasons:

Support companies shipping a Linux operating system pre-installed on their laptops and workstations
Hard proof Linux works on their hardware

After using Ubuntu for a few weeks I switched to NixOS. I will be posting about my experience with NixOS on the Z13 in a future blog post and will likely livestream about it on Twitch as well. TL;DR NixOS 22.11 has fewer issues than Ubuntu 22.04 on the ThinkPad Z13 Gen 1.

The ThinkPad Z13 Gen 1 is an excellent laptop for my requirements, and I have no regrets. If I were to buy I laptop pre-installed with a Linux distro in the future, I might go for Fedora, just so I can see what the OEM experience is like with Fedora.

Since I purchased the Z13, Framework has announced AMD 7040-series powered laptops are coming later in 2023; and I’m not sure I can resist…

Linux Matters

martin@wimpress.com (Martin Wimpress) — Tue, 18 Apr 2023 00:00:00 +0000

Join 3 experienced Open Source professionals (Alan Pope, Mark Johnson, Martin Wimpress) as they discuss the impact Linux has in their daily lives. Upbeat family-friendly banter, conversation and discussion for Linux enthusiasts and casual observers of all ages. A new episode every two weeks covering terminal productivity, desktop experience, development, gaming, hosting, hardware, community, cloud-native and all the Linux Matters that matter.

Organisation: Linux Matters
Date: April 2023 - date
Role: Writer & Presenter

machinespawn

martin@wimpress.com (Martin Wimpress) — Wed, 19 Oct 2022 00:00:00 +0000

machinespawn is a wrapper for machinectl and systemd-nspawn to creating and managing containers; primarily focused at local development environment and CI/CD with a heavy emphasis on caching.

Organisation: machinespawn
Date: October 2022 - date
Role: Project Lead

Linux Downtime

martin@wimpress.com (Martin Wimpress) — Sun, 24 Apr 2022 00:00:00 +0000

Open source professionals Martin Wimpress, Hayden Barnes, Gary Kramlich, and Alan Pope join Joe Ressington for relaxed discussions in their downtime. From working in the industry and progressing your career, to managing a project’s community, and beyond.

Organisation: Linux Downtime
Date: April 2022 - April 2023
Role: Co-presenter & Producer

deb-get

martin@wimpress.com (Martin Wimpress) — Mon, 18 Apr 2022 00:00:00 +0000

deb-get makes it easy to install and update .debs published in 3rd party apt repositories or made available via direct download on websites or GitHub release pages.

Organisation: deb-get
Date: April 2022 - date
Role: Project Lead

Creating Production-Ready Containers - Advanced Techniques

martin@wimpress.com (Martin Wimpress) — Fri, 18 Jun 2021 12:18:15 +0000

Creating production-ready containers for use in commercial-grade apps can be a far cry from the “get started with Node.js and Docker”-type of tutorials that are common around the Internet. Those guides are great for introducing all the advantages of Docker containers in modern cloud-native development, but creating a container that passes muster in a large-scale application in production is a different story.

For production-ready containers, there are three key things you want to optimise for when creating a container:

Image Size 📦
Build Speed 🐢
Security 🔐

Image size and build speed ensure that your containers can move through CI/CD and test pipelines easily and efficiently. Security is critical in today’s software supply chain, and containers have their own set of security issues. Thankfully, reducing container image size actually can alleviate some security issues in containers.

In my Basics article, I showed you some easy techniques to improve your Dockerfile using a sample “Hello World” Node.js application.

These basics address all three optimisations, though they only scratch the surface.

Let’s look at some more advanced techniques for Container Optimisation.

Alpine Images

The very first thing you’ll encounter when looking for techniques to create smaller containers is Alpine Linux. Alpine Linux is an open-source project whose goal is to create a bare-bones 🦴 version of Linux that lets developers “build from the ground up.”

Pros: Transitioning to Alpine can be an easy way to get a smaller container

Reducing image size with Alpine can be incredibly simple - under the right circumstances. For some apps, it’s as easy as changing the base image in your Dockerfile:

FROM

FROM node:16.2.0

TO

FROM node:16.2.0-alpine

When we build the new image, we see that the old image was 856MB and the new one is 114MB 🎉

REPOSITORY TAG IMAGE ID CREATED SIZE
cotw-node-alpine latest 2cc7b4a7b09c 2 minutes ago 114MB
cotw-node latest 873fb9fca53a 3 days ago 856MB

Easy, right? Not so fast.

Cons: Using Alpine images can lead to build problems, now and in the future

There are some not-so-obvious gotchas with using Alpine images that don’t crop up in our super simple example application, such as:

You have to install everything

Those tiny base images have to sacrifice something, right? Alpine users will be installing everything they need, right down to time-zone data or development tools. You won’t need your development tools for your production image, most likely, but for most developers, the thought of a server without curl or vim is a bridge too far.

Different compilers and package managers

You’ll also be installing any dependencies with the Alpine Package Keeper tool (apk) instead of the more familiar apt or rpm. The differences are small but can trip up unsuspecting developers.

Fewer examples; less documentation

Finally, while Alpine has been around for nine-plus years, it is and likely always will be a smaller and more specialised user base than established Linux distributions such as Ubuntu and Debian. To wit, at the time of this writing the alpine tag on StackOverflow has just 1,280 questions, compared with over 54,000 for Ubuntu.

Multi-stage builds

The next tactic you are likely to encounter when searching for ways to reduce Docker image sizes is multi-stage 🏗 builds. This tactic, recommended by Docker and many in the Docker community, is essentially building the image twice. The first set of commands builds your base application image, all things included. The second set of commands builds an image off of that base image, taking only what’s needed and leaving out anything that’s not.

With a multi-stage build, our Dockerfile would look like this. Notice the two FROM statements. The first builds the application image; the second copies the necessary files from that image into the second, more production-ready version.

FROM node:16.2.0-alpine as builder
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci
COPY app.js ./

FROM node:16.2.0-alpine
WORKDIR /usr/src/app
COPY --from=builder /usr/src/app .
EXPOSE 3000
USER node
CMD ["node","app.js"]

Pros: Dev and Prod images can be built separately

When combined with Docker Compose, this approach gives developers a flexible development environment while reducing bloat in the production images. You can simply use your initial image for dev/test and the final version for production. Multi-stage builds work especially well for Go containers, significantly reducing image size, but also work well for static Node.js and React-type applications.

Cons: Added complexity; use-case specific

Multi-stage builds are still relatively new 🌱 on the scene. For most developers still new to containers, knowing what to copy over to the final production image and what to leave behind is a major barrier to entry. Further, this pattern can run into challenges.

Since we’re already using an Alpine image, the size savings are relatively minor for our “Hello World” example. You’d expect to see greater gains in a full-blown React or Vue application.

REPOSITORY TAG IMAGE ID CREATED SIZE
cotw-node-multistage latest 52bc33d14a87 3 minutes ago 114MB
cotw-node-alpine latest 2cc7b4a7b09c 4 days ago 114MB
cotw-node latest 873fb9fca53a 7 days ago 856MB

Development tools and Distroless

There are several tools - and new ones emerging every day - that look to bypass or automate Dockerfile authoring to make image creation easier. Buildpacks are the most mature of these technologies and can be used through tools like Pack or Waypoint.

There are builder options from multiple sources - Heroku, Google, and Paketo are common favourites - and each gives you a slightly different developer experience and final image when used.

$ pack build cotw-node-bp-google --builder gcr.io/buildpacks/builder:v1
$ pack build cotw-node-bp-heroku --builder heroku/buildpacks:18
$ pack build cotw-node-bp-pb-base --builder paketobuildpacks/builder:base
$ pack build cotw-node-bp-pb-full --builder paketobuildpacks/builder:full

Pros: When they work, they work

In certain instances, Buildpacks can take the pain out of Dockerfile authoring and just create container images of your application with no fuss. The pack tool is looking for “app-like” files in your source directory, and automatically figuring out what kind of application is there and how to containerize it. In the case of our Node sample, it sees package.json and correctly assumes we have a Node.js application.

Cons: When they don’t…

Given the relative newness of this approach for Docker containers, there are a lot of gotchas with Buildpacks. Non-standard applications or operating systems can struggle, and we’ve had issues running them successfully on the new Silicon Macbook Pros. The resulting images vary a lot - we saw a range of 200MB to 800MB in our examples - and the results tend to be lower than what you’d get with other techniques.

Automate it with SlimToolKit

The SlimToolKit (formerly DockerSlim) open-source project was created by Slim.AI CTO Kyle Quest in 2015 as a way to automate container optimisation. Simply download and run slim build <myimage> and SlimToolKit will examine the image, rebuild it with only the required dependencies, and give you a new image that can be run just like the original.

Pros: It’s automatic

SlimToolKit means you can work with whatever base image you’d like (say, Ubuntu or Debian) and let SlimToolKit worry about removing unnecessary tools and files en route to production. The best part is that SlimToolKit can be used alongside any of these other techniques. Once tested, it can be integrated into your CI/CD pipeline for automatic container minification, and the reduction in size leads to faster build times and better security.

Cons: Steep learning curve

As with any open-source software, SlimToolKit can take some time to get working, especially for non-trivial applications. It works best for web-style applications, micro-services and APIs that have defined HTTP/HTTPS ports which the sensor can find and use to observe the container internals.

For best results, spend some time getting to know the various command flags available to tune your image, and take a look at the examples for whatever framework you’re using.

Connecting with SlimToolKit

There’s an active Slim.AI Discord channel full of experts who can help you triage issues as they arise.

Creating Production-Ready Containers - The Basics

martin@wimpress.com (Martin Wimpress) — Thu, 03 Jun 2021 12:18:15 +0000

So you’ve coded an awesome app and you are ready to deploy it to the cloud. You’ve heard a lot about Docker and completed a few online tutorials to containerise your app. All set, right? But what do you need to know if you’re going to move that app to a production environment on the public Internet? What if you’re using it for your job and need to pass security scans and DevOps checks?

In this series, I introduce some basic concepts for making production-ready containers. I also introduce the concept of “slimming” a container. Slimming refers to both optimising and minifying your Docker containers, reducing them in size by up to 80-percent while also making them more secure by decreasing the attack surface. Slimming your container is also a great way to implement container best practices without re-engineering your entire workflow.

There are many ways to slim a container, from basic security to fully automated open-source tools like SlimToolKit (formerly DockerSlim). Full disclosure: I used work for Slim.AI, a company founded on the SlimToolKit open source project. Let’s look at some of the common ways developers create production-ready container images today.

I’ll explore each of these in a separate article using a simple “Hello World” Node.js example that can be found in many online tutorials.

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
 res.send('Hello World!')
})

app.listen(port, () => {
 console.log(`Example app listening at http://localhost:${port}`)
})

Let’s get started by simply improving your Dockerfile to build a better Docker image.

Creating a Better Dockerfile

Most Dockerfile examples you’ll find are not “production ready” and they aren’t meant to be. They are for instructional purposes to help developers successfully build an image. But when one gets into production scenarios, there are several “good-to-know” and a few “have-to-know” techniques that will improve build times, security, and reliability.

Let’s look at a typical example that you might run into if you’re a Node.js developer looking to get “Hello World” running with Docker. I won’t go through building an actual app - there are a lot of great examples out there to show you how to do this - but rather focus on what to do if you were going to ship this to production.

The typical Dockerfile in a “Hello World” example might look something like this:

FROM node:latest
WORKDIR /usr/src/app
COPY package*.json app.js ./
RUN npm install
EXPOSE 3000
CMD ["node", "app.js"]

It uses the latest version of the official Node.js image, sets a directory and copies your app into the container image, installs dependencies, exposes port 3000, and runs the app via CMD.

While this will run no problem on your local machine, and is great for learning the ropes, this approach is almost certainly going to run into issues when you ship it to production. Let’s take a look at some of these in order of severity.

Major issues

Running as Root

Since this example doesn’t set a USER explicitly in the Dockerfile, Docker runs the build and all commands as the root user. While not an issue for local development, your friendly neighbourhood SysAdmin will tell you the myriad of issues that come with running applications as root on a server in production. And with Docker, a new set of attack methods can arise.

Thankfully, most major languages and frameworks have a predefined user for running applications. In Node.js, the user is just node and can be invoked in the Dockerfile explicitly.

FROM node:latest
WORKDIR /usr/src/app
COPY package*.json app.js ./
RUN npm install

USER node

EXPOSE 3000
CMD ["node", "app.js"]

Using `latest` version

Choosing a version number for your container is often called pinning. While many tutorials - and even some experts - will counsel newcomers to pin their images to the latest tag, which means you get whatever the most recently updated version is, using the latest tag can cause issues in production.

Containers are meant to be ephemeral, meaning they can be created, destroyed, started, stopped, and reproduced with ease and reliability. Using the latest tag means there isn’t a single source of truth for your container’s “bill of materials”. A new version or update of a dependency could introduce a breaking change, which may cause the build to fail somewhere in your CI/CD pipeline.

Example Dockerfile

FROM node:latest

Production Dockerfile

FROM node:16.2.0

Other tutorials I’ve seen pin only the major version. For example, using node:14. This carries the same risks as using latest, as minor versions can change dependencies as well.

Now, pinning a specific major and minor version in your Dockerfile is a trade-off decision - you’re choosing to not automatically receive security, fixes or performance improvements that come via new updates - but most DevSecOps teams prefer to employ security scanning and container management software as a way to control updates rather than dealing with the unpredictability that comes with container build failures in production CI/CD pipelines.

Performance improvements

Better layer caching

Docker works on the concept of layer caching. It builds images sequentially. Layering dependencies on top of each other and only rebuilding them when something in the layer has changed.

Layer 0 in a Docker image is often the base operating system, which rarely changes significantly; although commercial Linux vendors often publish new base images to incorporate security fixes.

Application code, however, is highly likely to change during the software development cycle, as you iterate on features, refactor, and fix bugs. Dependencies in our core system, installed here by npm install, change more often than the base OS, but less often than the application code.

In our example Dockerfile, we simply need to break the installation of the dependencies into separate instructions on their own lines.

FROM node:16.0.2
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci

USER node

COPY app.js ./

EXPOSE 3000
CMD ["node", "app.js"]

We actually end up creating another layer by now having two COPY commands. While adding layers is typically a no-no for build times and image sizes, the tax we pay on this optimisation is going to save us in the long run as we cycle through the QA process, since we aren’t reinstalling dependencies if we don’t have to.

We also opt for the npm ci command instead of npm install, which is preferred for automated environments, such as CI/CD, and will help prevent breaking changes from dependencies. Read more about npm ci here.

Use `ENTRYPOINT` instead of `CMD`

At a surface level, there isn’t a big difference between using ENTRYPOINT with your app file versus running CMD using the shell plus your app file. However, web- and API-type containers like Node.js applications are often running as executables in production, and there, proper signal handling - such as graceful shutdowns - are important.

CMD provides some flexibility for calling executables with flags or overwriting them, which is common in development. But that generally won’t be relevant to production instances and ENTRYPOINT will likely provide better signal processing.

FROM node:16.0.2
WORKDIR /usr/src/app

COPY package*.json ./
RUN npm ci

USER node

COPY app.js ./

EXPOSE 3000
ENTRYPOINT ["node", "app.js"]

Cleaning up cached files

Most package managers have the ability to clean up their own cache. If you don’t do this, you’ll just be moving a bunch of unused files into your container for no reason. It might not save a lot of space depending on your application, but think of it as dropping your unused items at the charity shop before you move rather than loading them in the moving van. It’s not a lot of effort and it’s the right thing to do. We do this by adding && npm cache clean --force to our RUN instruction.

FROM node:16.0.2
WORKDIR /usr/src/app
COPY package*.json ./

RUN npm ci && npm cache clean --force

USER node

COPY app.js ./

EXPOSE 3000
ENTRYPOINT ["node", "app.js"]

Conclusions

Improving your Dockerfile is the first step towards creating a slimmed and optimised container. It closes some major security loopholes that are likely to raise flags with downstream checks and adds baseline optimisations for build time and docker image size.

If this is all you do to improve your containers prior to shipping to production, you won’t be in a bad spot, but there’s more - way more - that you can do to optimise images. We’ll explore those techniques in the next article.

Retro Home

martin@wimpress.com (Martin Wimpress) — Tue, 21 Jul 2020 00:00:00 +0000

Retro Home is custom Raspberry Pi operating system purpose built for retro gaming. Built with Ubuntu and using the minimalist emulator frontend Ludo it supports most classic consoles from Atari, Nintendo, Sega and SNK along with arcade emulation support. A number of retro styled Raspberry Pi cases from Retroflag and Waveshare are supported (with more in the works) using bespoke GPIO drivers.

Organisation: Retro Home
Date: July 2020 - date
Role: Project Lead

Rolling Rhino

martin@wimpress.com (Martin Wimpress) — Sat, 27 Jun 2020 00:00:00 +0000

Simple shell script to convert Ubuntu into a “rolling release” that tracks the devel series; for the toughest of Ubuntu users.

This project has served its purpose. It inspired another group of developers to create a fully fledged Ubuntu Rolling Rhino distro.

Organisation: Rolling Rhino
Date: March 2020 - November 2022
Role: Project Lead

Quickemu

martin@wimpress.com (Martin Wimpress) — Sun, 15 Mar 2020 00:00:00 +0000

Quickly create and run highly optimised desktop virtual machines for Linux, macOS and Windows; with just two commands. You decide what operating system you want to run and Quickemu will figure out the best way to do it for you

Organisation: Quickemu Project
Date: March 2020 - date
Role: Project Lead

DIY SNES Classic

martin@wimpress.com (Martin Wimpress) — Tue, 24 Jan 2017 12:00:00 +0000

Inspired by the recent NES Classic I made a DIY SNES Classic just in time for the Christmas holidays and it’s very portable!

Raspberry Pi 3 in a 3D printed SNES case with 8Bitdo SNES30 controllers

To make one yourself you’ll need:

Raspberry Pi 3 Model B.
Retropie 4.1 or newer.
Two 8Bitdo SNES30 wireless controllers.
A SNES 3D Printed case.
CRT scanline shaders to complete the retro look on the big screen.

Both controllers use Bluetooth, so two player wire-free gaming is possible. The USB cables are just for charging, but if you’ve got no charge they can be used as wired controllers too. Retropie can be controlled via the controllers, no keyboard/mouse required.

Raspberry Pi 3 Nextcloud Box running on Ubuntu Core

martin@wimpress.com (Martin Wimpress) — Tue, 13 Dec 2016 17:17:00 +0000

I recently bought the Nextcloud Box. When it came to setting it up I ran into a problem, I only had Raspberry Pi 3 computers available and at the time of writing the microSDHC card provided with the Nextcloud Box only supports the Raspberry Pi 2. Bummer!

Overview

This guide outlines how to use Ubuntu Core on the Raspberry Pi 3 to run Nextcloud provided as a snap from the Ubuntu store.

If you’re not familiar with Ubuntu Core, here’s a quote:

Ubuntu Core is a tiny, transactional version of Ubuntu for IoT devices and large container deployments. It runs a new breed of super-secure, remotely upgradeable Linux app packages known as snaps

After following this guide Ubuntu Core and any installed snaps (and their data) will reside on the SD card and the 1TB hard disk in the Nextcloud box will be available for file storage. This guide explains how to:

Install and configure Ubuntu Core 16 for the Raspberry Pi 3
Format the 1TB hard disk in the Nextcloud Box and auto-mount it
Install the Nextcloud snap and connect the removable-media interface to allow access to the hard disk
Activate and configure the Nextcloud External Storage app so the hard disk can be used to store files
Optional configuration of Email and HTTPS for Nextcloud

Prepare a microSDHC card

I explained the main steps in this post but you really should read and follow the Get started with a Raspberry Pi 2 or 3 page as it fully explains how to use a desktop computer to download an Ubuntu Core image for your Raspberry Pi 2 or 3 and copy it to an SD card ready to boot.

Here’s how to create an Ubuntu Core microSDHC card for the Raspberry Pi 3 using an Ubuntu desktop:

Download Ubuntu Core 16 image for Raspberry Pi 3
Insert the microSDHC card into your PC
- Use GNOME Disks and its Restore Disk Image… option, which natively supports XZ compressed images.
- Select your SD card from the panel on the left
- Click the “burger menu” on the right and Select Restore Disk Image…
- Making sure the SD card is still selected, click the Power icon on the right.
Eject the SD card physically from your PC.

Ubuntu Core first boot

An Ubuntu SSO account is required to setup the first user on Ubuntu Core:

Start by creating an Ubuntu SSO account
Import an SSH Key into your Ubuntu SSO account
- Here are instructions to generate an SSH Key
You’ll need a keyboard and monitor connected to the Raspberry Pi 3 to go complete the first boot process and device configuration.

Insert the Ubuntu Core microSHDC into the Raspberry Pi, which should be in the assembled Nextcloud Box with a keyboard and monitor connected. Plug in the power.

The system will boot then become ready to configure
The device will display the prompt “Press enter to configure”
Press enter then select “Start” to begin configuring your network and an administrator account. Follow the instructions on the screen, you will be asked to configure your network and enter your Ubuntu SSO credentials
At the end of the process, you will see your credentials to access your Ubuntu Core machine:

This device is registered to <Ubuntu SSO email address>.
Remote access was enabled via authentication with the SSO user <Ubuntu SSO user name>
Public SSH keys were added to the device for remote access.

Once setup is done, you can login to Ubuntu Core using ssh, from a computer on the same network, using the following command:

ssh <Ubuntu SSO user name>@<device IP address>

The user name is your Ubuntu SSO user name.

Reconfiguring network

Should you need to reconfigure the network at a later stage you can do so with:

sudo console-conf

Prepare 1TB hard disk

ssh <Ubuntu SSO user name>@<device IP address>

Partition and format the Nextcloud Box hard disk

This will create a single partition formatted with the ext4 filesystem.

sudo fdisk /dev/sda

Do the following to create the partition:

Command (m for help): o
Created a new DOS disklabel with disk identifier 0x253fea38.

Command (m for help): n
Partition type
 p primary (0 primary, 0 extended, 4 free)
 e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-1953458175, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-1953458175, default 1953458175):

Created a new partition 1 of type 'Linux' and of size 931.5 GiB.

Command (m for help): w

Now format the partition and give it the label data. This label will be used to reference it for mounting later:

sudo mkfs.ext4 -L data /dev/sda1

Automatically mount the partition

Most of the Ubuntu Core root file system is read-only, so it is not possible to edit /etc/fstab. Therefore we’ll use systemd to achieve that.

Be aware of one of the systemd.mount pitfalls:

Mount units must be named after the mount point directories they control. Example: the mount point /home/lennart must be configured in a unit file home-lennart.mount.

Yes that’s right! The unit filename must match the mount point path.

Create the media-data.mount unit:

sudo vi /writable/system-data/etc/systemd/system/media-data.mount

Add the following content:

[Unit]
Description=Mount unit for data

[Mount]
What=/dev/disk/by-label/data
Where=/media/data
Type=ext4

[Install]
WantedBy=multi-user.target

Reload systemd, scanning for new or changed units:

sudo systemctl daemon-reload

Start the media-data.mount unit, which will mount the volume, and also enable it so it will be automatically mounted on boot.

sudo systemctl start media-data.mount
sudo systemctl enable media-data.mount

And just like any other unit, you can view its status using systemctl status:

sudo systemctl status media-data.mount

Update Ubuntu Core

Make sure Ubuntu Core is up-to-date and reboot.

sudo snap refresh
sudo reboot

After the reboot, make sure /media/data is mounted. If not double check the steps above.

Install Nextcloud

The Nextcloud snap uses the removable-media interface, which grants access to /media/*, and requires manual connection:

sudo snap install nextcloud
sudo snap connect nextcloud:removable-media core:removable-media

Browse to the Nextcloud IP address and create the admin user account, for example:

http://nextcloud.local/

Nextcloud configuration

In the examples below replace nextcloud.local with the IP address or hostname of your Nextcloud Box and replace example.org with your domain.

External Storage

Enable the External Storge app via:

http://nextcloud.local/index.php/settings/apps?category=disabled#

Configure External Storage app via:

http://nextcloud.local/index.php/settings/admin/externalstorages

Use these settings:

Folder name: data
External storage: Local
Authentication: None
Configuration: /media/data
Available for: All

Email

Configure your outgoing email settings via:

http://nextcloud.local/index.php/settings/admin/additional

I use Sendgrid for sending email alerts from my servers and devices. These are the settings that work for me:

Send mode: SMTP
Encryption: STARTTLS
From address: nextcloud@example.org
Authentication method: Plain
Authentication required: Yes
Server address: smtp.sendgrid.net:587
Username: apikey
Password: theactualapikey

Enabling HTTPS

It is strongly recommend that you use HTTPS if you intend to expose your Nextcloud to the Internet.

First do a test to see if you can install a Let’s Encrypt certificate:

sudo nextcloud.enable-https -d

Answer the questions:

Have you met these requirements? (y/n) y
Please enter an email address (for urgent notices or key recovery): name@example.org
Please enter your domain name(s) (space-separated): nextcloud.example.org
Attempting to obtain certificates... done
Looks like you're ready for HTTPS!

If everything went well, then install the certificate

sudo nextcloud.enable-https

Answer the questions again:

Have you met these requirements? (y/n) y
Please enter an email address (for urgent notices or key recovery): name@example.org
Please enter your domain name(s) (space-separated): nextcloud.example.org
Attempting to obtain certificates... done
Restarting apache... done

If Let’s Encrypt didn’t work for you, you can always use Nextcloud with a self-signed certificate.

sudo nextcloud.enable-https -s

Manual configuration changes

If you need to make any tweaks to the Nextcloud configuration file you can edit it like so:

sudo vi /var/snap/nextcloud/current/nextcloud/config/config.php

If you have manually edited the Nextcloud configuration you may need to restart nextcloud:

sudo snap disable nextcloud
sudo snap enable nextcloud

Conclusion

So there it is, Nextcloud running on Ubuntu Core powered by a Raspberry Pi 3. The performance is reasonable, obviously not stellar, but certainly good enough to move some cloud services for a small family away from the likes of Google and Dropbox. Now go and install some Nextcloud clients for your desktops and devices :-)

HP Microserver N54L power saving and performance tuning using Debian

martin@wimpress.com (Martin Wimpress) — Wed, 16 Dec 2015 12:00:00 +0000

I’ve installed Open Media Vault on a HP ProLiant MicroServer G7 N54L and use it as media server for the house. OpenMediaVault (OMV) is a network attached storage (NAS) solution based on Debian.

I want to minimise power consumption but maximise performance. Here are some tweaks reduce power consumption and improve network performance.

Power Saving

Install the following.

apt-get install amd64-microcode firmware-linux firmware-linux-free \
firmware-linux-nonfree pciutils powertop radeontool

And for ACPI.

apt-get install acpi acpid acpi-support acpi-support-base

ASPM and ACPI

First I enabled PCIE ASPM in the BIOS and forced the kernel to use it and ACPI via grub by changing GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, so it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet acpi=force pcie_aspm=force nmi_watchdog=0"

Then update grub and reboot.

update-grub
reboot

Enable Power Saving via udev

The following rules file /etc/udev/rules.d/90-local-n54l.rules enables power saving modes for all PCI, SCSI and USB devices and ASPM. Futher the internal Radeon card power profile is set to low as there is rarely a monitor connected. The file contains the following:

SUBSYSTEM=="module", KERNEL=="pcie_aspm", ACTION=="add", TEST=="parameters/policy", ATTR{parameters/policy}="powersave"
SUBSYSTEM=="i2c", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="pci", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/autosuspend", ATTR{power/autosuspend}="2"
SUBSYSTEM=="scsi", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="spi", ACTION=="add", TEST=="power/control", ATTR{power/control}="auto"
SUBSYSTEM=="drm", KERNEL=="card*", ACTION=="add", DRIVERS=="radeon", TEST=="power/control", TEST=="device/power_method", ATTR{device/power_method}="profile", ATTR{device/power_profile}="low"
SUBSYSTEM=="scsi_host", KERNEL=="host*", ACTION=="add", TEST=="link_power_management_policy", ATTR{link_power_management_policy}="min_power"

Add this to /erc/rc.local.

echo '1500' > '/proc/sys/vm/dirty_writeback_centisecs'

Hard disk spindown

Using the Open Media Vault web interface got to Storage -> Physical Disks, select each disk in turn and click Edit then set:

Advanced Power Management: Intermediate power usage with standby
Automatic Acoustic Management: Minimum performance, Minimum acoustic output
Spindown time: 20 minutes

Performance Tuning

Network

The following tweaks improve network performance ,but I have a HP NC360T PCI Express Dual Port Gigabit Server Adapter in my N54L so these settings may not be applicable to the onboard NIC.

Add this to /erc/rc.local.

ethtool -G eth0 rx 4096
ethtool -G eth1 rx 4096
ethtool -G eth0 tx 4096
ethtool -G eth1 tx 4096
ifconfig eth0 txqueuelen 1000
ifconfig eth1 txqueuelen 1000

Add the following to /etc/sysctl.d/local.conf.

fs.file-max = 100000
net.core.netdev_max_backlog = 50000
net.core.optmem_max = 40960
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_sack=0
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
vm.swappiness = 10

Conclusion

With these settings applied powertop reports everything that can be in a power saving mode is and the room temperature is measurably cooler. More importantly, with four 4TB drives in a RAID-5 configuration formatted with XFS and dual bonded gigabit ethernet, I am able to backup data to the server at a sustained rate of 105MB/s, which is 0.85 Gbit.

Not too shabby for an AMD Turion II Neo N54L (2.2GHz) 🙂

References

Intel NUC5i7RYH with Ubuntu

martin@wimpress.com (Martin Wimpress) — Sun, 13 Dec 2015 13:18:00 +0000

I’ve just bought my first brand new computer since 2008. Thanks to the Black Friday and Cyber Monday sales on Amazon.co.uk and Scan.co.uk this year I was able to put together a pretty sweet Intel NUC which is now running Ubuntu MATE 15.10.

I spoke about this new system on LINUX Unplugged Episode 122 and have been contacted by people wanting more details. Hopefully this blog post will answer any outstanding questions. Press play below to hear to what I said on the podcast.

The NUC

I purchased an Intel Next Unit of Computing (NUC) Kit NUC5I7RYH.

Barebones mini PC
1 x Core i7 5557U / 3.1 GHz
Iris Graphics 6100
Gigabit Ethernet
Bluetooth 4.0 LE
802.11a/b/g/n/ac WiFi

Everything works out of the box with Ubuntu MATE 15.10 including sending audio to the monitor over HDMI and DisplayPort.

The RAM

The Core i7 5557U specs state that DDR3L 1866 Mhz RAM is supported. The arstechnica Mini-review: Intel’s powered-up Core i7 Broadwell mini PC has some benchmarks that show a performance improvement when using 1866 Mhz clocked RAM, so I purchased:

HyperX Impact SODIMM - 16GB Kit*(2x8GB) - DDR3L 1866MHz CL11 SODIMM

The SSDs

I have two SSDs in the NUC.

Samsung MZVPV256HDGL-00000 - SM951 256GB M.2 PCI-e 3.0 x 4 NVMe Solid State Drive
SanDisk Ultra II SSD 960 GB Sata III 2.5-inch Internal SSD

This is where you might need to do some more research. At the time of writing, it is not possible to boot directly from the Samsung SM951 NVMe SSD. Although other quad channel NVMe SSDs do appear to be supported as the boot device. It is possible that the Samsung SSD 950 Pro, which is also NVMe, can be used as the boot device in Linux, but don’t take my word for it. Do your research.

My work around was make the SanDisk Ultra II the boot device by putting /boot and the MBR on it. I have put / and swap (swap because I want to experiment with suspend and hibernate) are on the Samsung SM951 and /home is on the SanDisk Ultra II.

There are faster SSDs than the SanDisk Ultra II but this was 50% off during Black Friday and just too good a deal to pass by. I did sacrifice a little performance on this component, so if absolute performance is your goal look at alternative SSDs, the Samsung 850 EVO seems to benchmark favourably.

The Monitor

A week after I purchased the NUC I noticed ebuyer.com were selling the monitor I wanted with a £100 discount. So I snapped one up.

Samsung S27D850T 27" WQHD LED DVI HDMI Monitor

The NUC is connected via mini Display Port and my Dell Precision T7400 is connected via HDMI. The Samsung S27D850T has an audio out and a 2.1 speaker set is connected to it.

Conclusion

I’m extremely happy with the NUC5I7RYH. Linux compatibility is first class, it’s fast and has relatively low power consumption. You can even charge a mobile phone using one of the front USB ports (the yellow one) when the NUC is powered off. It is my principle workstation and is able to handle everything I demand from it, with ease:

mp3 and ogg audio encoding
h.264 video encoding (~1 min of video @480p encodes in 1 second)
running multiple virtual machines
compiling large applications
creating xz compressed images of Ubuntu flavours for the Raspberry Pi 2

I’ve not tried gaming on it yet, but the holidays are approaching and GRID Autosport was released for SteamOS this week. So I know what I’ll be doing in a couple of weeks time :-D

Installing Nikola on Ubuntu

martin@wimpress.com (Martin Wimpress) — Thu, 05 Nov 2015 11:00:00 +0000

Nikola is a static site and blog generator written in Python that I’ve been using for a good while now. This blog post describes how to install Nikola on Ubuntu 14.04 or newer. Now, this may look like a long winded way to install Nikola, given that .deb package exists, but in my opinion it is the correct way to install Nikola on Ubuntu.

Installing Python

First you’ll need Python.

sudo apt-get install cython3 libpython3.4 python3.4 python3.4-dev python3.4-minimal

Now install the Python “package” management utilities.

sudo apt-get install python-setuptools python-virtualenv python-pip virtualenvwrapper

The Snakepit

Create a “Snakepit” directory for storing all the virtualenvs.

mkdir ~/Snakepit

Create a virtualenv for Nikola

The following will create a new virtualenv called nikola based on Python 3.4.

virtualenv -p /usr/bin/python3.4 ~/Snakepit/nikola-773

Working on a virtualenv

To activate the virtualenv do the following.

source ~/Snakepit/nikola-773/bin/activate

Your shell prompt will change while a virtualenv is being worked on to indicate which virtualenv is currently active.

While working on a virtualenv you can pip install what you need or manually install any Python libraries safe in the knowledge you will not adversely damage any other virtualenvs or the global packages in the process. Very useful for developing a new branch which may have different library requirements than the master/head.

When you are finished working in a virtualenv you can deactivate it by simply executing deactivate.

Install Nikola requirements

Nikola requires some additional packages.

sudo apt-get install liblcms2-dev libfreetype6-dev libjpeg8-dev \
libopenjp2-7-dev libtiff5-dev libwebp-dev libxslt1-dev libxml2-dev \
libyaml-dev libzmq-dev zlib1g-dev

Some of the content optimisation filters require additional packages.

sudo apt-get install closure-compiler jpegoptim optipng yui-compressor

Install Tidy 5. (optional)

sudo apt-get -y remove libtidy-0.99-0 tidy
wget http://binaries.html-tidy.org/binaries/tidy-5.1.14/tidy-5.1.14-64bit.deb -O /tmp/tidy5.deb
sudo dpkg -i /tmp/tidy5.deb
sudo ln -s /usr/bin/tidy /usr/local/bin/tidy5
rm /tmp/tidy5.deb

What are these requirements for?

The following are required to build pillow, the Python imaging library.

liblcms2-dev
libfreetype6-dev
libjpeg8-dev
libopenjp2-7-dev
libtiff5-dev
libwebp-dev
zlib1g-dev

The following are required to build lxml, a Python XML library.

libxml2-dev
libxslt1-dev

The following are required to build python-coveralls.

libyaml-dev

The following are required to build pyzmq.

libzmq-dev

Install Nikola

First install Cython, which will ensure some of the packages required by Nikola use all the available optimisations.

pip install --upgrade Cython

Install all of Nikola.

pip install --upgrade "Nikola[extras,tests]"

Create a site

After installing Nikola, you should create a site. A site is a collection of all assets needed to render your website, including configuration, posts, pages, images, and all other files and customizations.

To create a site, you need to run:

nikola init <directory_name>

A wizard will guide your initial setup The --demo option can be used to populate your site with some example content. If you do not want the wizard, use the --quiet argument.

Nikola is now installed and and initial site is setup. nikola help and the Nikola Handbook will assist you from here.

Ubuntu Podcast

martin@wimpress.com (Martin Wimpress) — Sun, 01 Feb 2015 00:00:00 +0000

Ubuntu Podcast was a weekly podcast that discussd news from the Ubuntu and Open Source community. The podcast ran from 2008 to 2021 and I joined the line up of presenters in 2015. In addition to the weekly podcasts we also participated in live shows at OggCamp and FOSS Talk Live.

Organisation: Ubuntu Podcast
Date: February 2015 - September 2021
Role: Co-presenter & Producer

Debian

martin@wimpress.com (Martin Wimpress) — Wed, 01 Oct 2014 00:00:00 +0000

Debian is a Unix-like computer operating system and a Linux distribution that is composed entirely of free and open-source software, most of which is under the GNU General Public License, and packaged by a group of individuals known as the Debian Project.

Organisation: Debian Project
Date: October 2014 - date
Role: Package maintainer

Installing Willie IRC Bot on Debian

martin@wimpress.com (Martin Wimpress) — Sat, 09 Aug 2014 11:11:11 +0000

Willie is an IRC bot written in Python that I’ve recently started using. This blog post describes how to install Willie on Debian and as usual I will be using virtualenv to isolate this Python application from the rest of the system.

Installing Python

First you’ll need Python.

sudo apt-get install libpython2.7 python2.7 python2.7-dev python2.7-minimal

The following will also be required to enable all the features Willie supports.

sudo apt-get install enchant python2.7-dev libxslt1-dev libxml2-dev

Remove any apt installed Python packages that we are about to replace. The versions of these packages in the Debian repositories soon get stale.

sudo apt-get purge python-setuptools python-virtualenv python-pip python-profiler

Install pip.

wget https://bootstrap.pypa.io/get-pip.py
sudo python2.7 get-pip.py

Use pip to install virtualenv.

sudo pip install virtualenv --upgrade

The Snakepit

Create a “Snakepit” directory for storing all the Python virtual environments.

mkdir ~/Snakepit

Create a virtualenv for Willie

The following will create a new virtualenv called willie using Python 2.7 as the interpreter.

virtualenv -p /usr/bin/python2.7 ~/Snakepit/willie

Working on a virtualenv

Activate the virtualenv for Willie.

source ~/Snakepit/willie/bin/activate

Your shell prompt will change, something like (willie)user@host:~$, while a virtualenv is being worked on to indicate which virtualenv is currently active.

While working on a virtualenv you can pip install what you need or manually install any Python libraries safe in the knowledge you will not upset any other virtualenvs or the global packages in the process. Very useful for developing a new branch which may have different library requirements than the current stable release.

When you are finished working in a virtualenv you can deactivate it by simply executing deactivate.

Install Willie

I’ve decided to use Python 2.7 to run Willie and therefore have to install backports.ssl_match_hostname which is not required if you use Python 3.3.

pip install willie backports.ssl_match_hostname

Additional functionality

Willie has no external dependencies, besides Python. However, some of the modules do have external dependencies. So install the following Python modules so that I can make use of everything Willie can do.

pip install feedparser pytz lxml praw pyenchant pygeoip ipython --upgrade

Configure Willie

I am not going to explain to how to configure Willie because all that good stuff is very well documented by the project.

https://github.com/embolalia/willie/wiki

But for reference, my default.cfg looks something like this:

[core]
nick = nicofyourbot
user = nicofyourbot
name = Give You Bot A Name
host = chat.freenode.net
use_ssl = true
verify_ssl = true
port = 6697
owner = nicofthebotowner
channels = #example
nickserv_password = ************
prefix = \.
timeout = 120

[db]
userdb_type = sqlite
userdb_file = /home/username/.willie/willie.db

Willie as a daemon

From this point on I assume you’ve completed the first run configuration of Willie and have .willie/default.cfg in your home directory.

Add the following to /etc/init.d/willie.

#!/bin/sh
### BEGIN INIT INFO
# Provides: willie
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Should-Start: $network
# Should-Stop: $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Willie IRC Bot.
# Description: Start and stops the Willie IRC bot for a given user.
### END INIT INFO

# NOTE! Replace with the user you want to run Willie.
willie_USER="yourusername"

HOMEDIR=$(getent passwd $willie_USER | awk -F: '{print $6}')
DAEMON="$HOMEDIR/Snakepit/willie/bin/willie"
CONFIG="$HOMEDIR/.willie/default.cfg"

startd() {
 if [ -f ${CONFIG} ]; then
 echo "Starting Willie for $willie_USER"
 start-stop-daemon -c $willie_USER -u $willie_USER -x $DAEMON -S -- --config ${CONFIG} --fork --quiet
 else
 echo "Couldn't start Willie for $willie_USER (no $CONFIG found)"
 fi
}

stopd() {
 echo "Stopping Willie for $willie_USER"
 willie_PID=$(pgrep -fu $willie_USER $DAEMON)
 if [ -z "$willie_PID" ]; then
 echo "Willie for USER $willie_USER: not running."
 else
 kill -15 $willie_PID
 fi
}

status() {
 willie_PID=$(pgrep -fu $willie_USER $DAEMON)
 if [ -z "$willie_PID" ]; then
 echo "Willie for USER $willie_USER: not running."
 else
 echo "Willie for USER $willie_USER: running (pid $willie_PID)"
 fi
}

case "$1" in
 start) startd ;;
 stop) stopd ;;
 restart|reload|force-reload) stopd && startd ;;
 status) status ;;
 *) echo "Usage: /etc/init.d/willie {start|stop|reload|force-reload|restart|status}"
 exit 1
 ;;
esac

exit 0

Set the permissions.

sudo chmod +x /etc/init.d/willie

Check that you can start/stop Willie.

sudo /etc/init.d/willie start
sudo /etc/init.d/willie status
sudo /etc/init.d/willie stop

Add willie to the startup/shutdown sequence.

sudo update-rc.d willie defaults

And that’s it. Willie is now running as a daemon inside a virtualenv.

ZNC IRC proxy

martin@wimpress.com (Martin Wimpress) — Sat, 02 Aug 2014 10:10:10 +0000

I have been using the BIP IRC proxy that maintains a persistent connection to a list of IRC channels. However, I’ve heard good things about ZNC and decided to give it a try.

The purpose of an IRC proxy, or bouncer, is that you can then point your IRC clients to them to maintain a transparent connection from multiple clients and playback the conversations that took place while you were away.

Installing ZNC

The ZNC package for Debian Wheezy are very old, so I decide to install from source.

Install required packages

We first need to make sure we have all the packages required to build ZNC.

sudo apt-get install build-essential libssl-dev libperl-dev pkg-config

Compile ZNC

Now download and compile ZNC.

cd
wget http://znc.in/releases/znc-1.4.tar.gz
tar zxvf znc-1.4.tar.gz
cd znc-1.4
./configure --with-openssl
make
sudo make install

Create a user

Create a separate ZNC user so that ZNC does not need to run as root:

sudo groupadd znc
sudo adduser --system --home /var/lib/znc --group znc

Configuring ZNC

You can use the interactive wizard to configure ZNC which really help create the initial configuration.

sudo -u znc /usr/local/bin/znc --datadir=/var/lib/znc --makeconf

Here is a transcript of how I answered the initial configuration questions.

[ .. ] Checking for list of available modules...
[ >> ] ok
[ ** ] Building new config
[ ** ]
[ ** ] First let's start with some global settings...
[ ** ]
[ ?? ] What port would you like ZNC to listen on? (1025 to 65535): 7778
[ ?? ] Would you like ZNC to listen using SSL? (yes/no) [no]: yes
[ ?? ] Would you like ZNC to listen using both IPv4 and IPv6? (yes/no) [yes]:
[ .. ] Verifying the listener...
[ >> ] ok
[ ** ] Unable to locate pem file: [/var/lib/znc/znc.pem], creating it
[ .. ] Writing Pem file [/var/lib/znc/znc.pem]...
[ >> ] ok
[ ** ]
[ ** ] -- Global Modules --
[ ** ]
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] | Name | Description |
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] | partyline | Internal channels and queries for users connected to znc |
[ ** ] | webadmin | Web based administration module |
[ ** ] +-----------+----------------------------------------------------------+
[ ** ] And 10 other (uncommon) modules. You can enable those later.
[ ** ]
[ ?? ] Load global module <partyline>? (yes/no) [no]: yes
[ ?? ] Load global module <webadmin>? (yes/no) [no]: yes
[ ** ]
[ ** ] Now we need to set up a user...
[ ** ]
[ ?? ] Username (AlphaNumeric): yournick
[ ?? ] Enter Password:
[ ?? ] Confirm Password:
[ ?? ] Would you like this user to be an admin? (yes/no) [yes]:
[ ?? ] Nick [yournick]:
[ ?? ] Alt Nick [yournick_]:
[ ?? ] Ident [yournick]:
[ ?? ] Real Name [Got ZNC?]: Your Name
[ ?? ] Bind Host (optional):
[ ?? ] Number of lines to buffer per channel [50]: 1024
[ ?? ] Would you like to clear channel buffers after replay? (yes/no) [yes]:
[ ?? ] Default channel modes [+stn]:
[ ** ]
[ ** ] -- User Modules --
[ ** ]
[ ** ] +--------------+------------------------------------------------------------------------------------------+
[ ** ] | Name | Description |
[ ** ] +--------------+------------------------------------------------------------------------------------------+
[ ** ] | chansaver | Keep config up-to-date when user joins/parts |
[ ** ] | controlpanel | Dynamic configuration through IRC. Allows editing only yourself if you're not ZNC admin. |
[ ** ] | perform | Keeps a list of commands to be executed when ZNC connects to IRC. |
[ ** ] | webadmin | Web based administration module |
[ ** ] +--------------+------------------------------------------------------------------------------------------+
[ ** ] And 21 other (uncommon) modules. You can enable those later.
[ ** ]
[ ?? ] Load module <chansaver>? (yes/no) [no]: yes
[ ?? ] Load module <controlpanel>? (yes/no) [no]: tes
[ ?? ] Load module <controlpanel>? (yes/no) [no]: yes
[ ?? ] Load module <perform>? (yes/no) [no]: yes
[ ?? ] Load module <webadmin>? (yes/no) [no]: yes
[ ** ]
[ ?? ] Would you like to set up a network? (yes/no) [no]:
[ ** ]
[ ?? ] Would you like to set up another user? (yes/no) [no]:
[ .. ] Writing config [/var/lib/znc/configs/znc.conf]...
[ >> ] ok
[ ** ]
[ ** ]To connect to this ZNC you need to connect to it as your IRC server
[ ** ]using the port that you supplied. You have to supply your login info
[ ** ]as the IRC server password like this: user/network:pass.
[ ** ]
[ ** ]Try something like this in your IRC client...
[ ** ]/server <znc_server_ip> +7778 flexiondotorg:<pass>
[ ** ]And this in your browser...
[ ** ]https://<znc_server_ip>:7778/
[ ** ]
[ ?? ] Launch ZNC now? (yes/no) [yes]: no

Running ZNV as a daemon

Here is a /etc/init.d/znc for Debian based on this installation method:

#! /bin/sh
### BEGIN INIT INFO
# Provides: znc
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: ZNC IRC bouncer
# Description: ZNC is an IRC bouncer
### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/bin
DESC="ZNC daemon"
NAME=znc
DAEMON=/usr/local/bin/$NAME
DATADIR=/var/lib/znc
DAEMON_ARGS="--datadir=$DATADIR"
PIDDIR=$DATADIR/run
PIDFILE=$PIDDIR/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
USER=znc
GROUP=znc

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
 # Return
 # 0 if daemon has been started
 # 1 if daemon was already running
 # 2 if daemon could not be started
 if [ ! -d $PIDDIR ]
 then
 mkdir $PIDDIR
 fi
 chown $USER:$GROUP $PIDDIR
 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test --chuid $USER > /dev/null || return 1
 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --chuid $USER -- $DAEMON_ARGS > /dev/null || return 2
}

#
# Function that stops the daemon/service
#
do_stop()
{
 # Return
 # 0 if daemon has been stopped
 # 1 if daemon was already stopped
 # 2 if daemon could not be stopped
 # other if a failure occurred
 start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME --chuid $USER
 RETVAL="$?"
 [ "$RETVAL" = 2 ] && return 2
 # Wait for children to finish too if this is a daemon that forks
 # and if the daemon is only ever run from this initscript.
 # If the above conditions are not satisfied then add some other code
 # that waits for the process to drop all resources that could be
 # needed by services started subsequently. A last resort is to
 # sleep for some time.
 start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON --chuid $USER
 [ "$?" = 2 ] && return 2
 # Many daemons don't delete their pidfiles when they exit.
 rm -f $PIDFILE
 return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
 start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME --chuid $USER
 return 0
}

case "$1" in
 start)
 [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
 do_start
 case "$?" in
 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
 stop)
 [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
 do_stop
 case "$?" in
 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
 status)
 status_of_proc -p $PIDFILE "$DAEMON" "$NAME" && exit 0 || exit $?
 ;;
 reload)
 log_daemon_msg "Reloading $DESC" "$NAME"
 do_reload
 log_end_msg $?
 ;;
 restart)
 log_daemon_msg "Restarting $DESC" "$NAME"
 do_stop
 case "$?" in
 0|1)
 do_start
 case "$?" in
 0) log_end_msg 0 ;;
 1) log_end_msg 1 ;; # Old process is still running
 *) log_end_msg 1 ;; # Failed to start
 esac
 ;;
 *)
 # Failed to stop
 log_end_msg 1
 ;;
 esac
 ;;
 *)
 echo "Usage: $SCRIPTNAME {status|start|stop|reload|restart}" >&2
 exit 3
 ;;
esac

After you’ve created the script, you must give it the proper permissions to run and add the script to the startup/shutdown sequence.

sudo chmod 755 /etc/init.d/znc
sudo update-rc.d znc defaults

Start the service:

sudo service znc start

Stop the service:

sudo service znc stop

Web configuration

I love that ZNC comes bundled with a web based configuration tool. Just login to https://znc.example.org:7778 to add users, add networks to users and to add channels to networks. Really simple stuff.

IRC client configuration

I use HexChat, but other IRC clients are available. Just add a new Network to HexChat for your ZNC server, use the username, suffixed with the network name you configured in ZNC, and your ZNC password.

Conclusion

I much prefer ZNC to BIP.

I really like the web and IRC configuration but I still have the option to configure the config files directly.
ZNC is far less cryptic with regard to setting up IRC client connections and user management is much better implemented.
When I add a new channel to an existing network in ZNC it automatically appears in my connected clients without the need to restart anything.
ZNC’s IRC backlogs don’t have the confusing double time stamps present in BIP and ZNC is much faster re-establishing connections to my multiple IRC network and channels.
Most importantly, ZNC has been far more stable than BIP.

References

Monitorix on Debian

martin@wimpress.com (Martin Wimpress) — Sat, 19 Jul 2014 12:00:00 +0000

I have a few Debian servers that run at home and on VPSs. I wanted to add some basic systems monitoring to them, but didn’t want anything too complicated to look after. I found Monitorix.

Monitorix is a free, open source, lightweight system monitoring tool designed to monitor as many services and system resources as possible. It has been created to be used under production Linux/UNIX servers, but due to its simplicity and small size can be used on embedded devices as well.

Install Monitorix

This install has been tested on Debian Squeeze and Wheezy. First install the dependencies.

sudo apt-get install rrdtool perl libwww-perl libmailtools-perl \
libmime-lite-perl librrds-perl libdbi-perl libxml-simple-perl \
libhttp-server-simple-perl libconfig-general-perl libio-socket-ssl-perl

Now Monitorix itself.

wget -c "http://apt.izzysoft.de/ubuntu/dists/generic/index.php?file=monitorix_3.5.1-izzy1_all.deb" -O monitorix_3.5.1-izzy1_all.deb
sudo dpkg -i monitorix_3.5.1-izzy1_all.deb

At this point Monitorix is installed and running. Point your browser to http://example.org:8080/monitorix/ and enjoy!

Configuring Monitorix

Everything in /etc/monitorix/monitorix.conf is comprehensively documented, just get tweaking.

http://www.monitorix.org/documentation.html

Each time you update the configuration Monitorix will require a restart.

sudo service monitorix restart

nginx status

If you run nginx then you’ll want to drop the following into /etc/nginx/conf.d/status.conf so that Monitorix can monitor nginx.

server {
 listen localhost:80;
 location /nginx_status {
 stub_status on;
 access_log off;
 allow 127.0.0.1;
 deny all;
 }
}

References

http://www.monitorix.org/doc-debian.html

subSonic on Debian

martin@wimpress.com (Martin Wimpress) — Sat, 12 Jul 2014 12:00:00 +0000

Last year I removed all my music from Google Play Music and created my own subSonic server. I really like subSonic but don’t use it a huge amount, mostly for syncing some music to my phone prior to going on holiday or business. Therefore, I’ve made a single one time donation to the project rather than the ongoing monthly usage fee.

Installing subSonic on Debian

This is how I install subSonic on Debian Wheezy.

Install Tomcat.

sudo apt-get install tomcat7

Install subSonic.

apt-get install ffmpeg
sudo mkdir /var/subsonic
sudo chown tomcat7: /var/subsonic
sudo wget -c https://github.com/KHresearch/subsonic/releases/download/v4.9-kang/subsonic.war
sudo cp subsonic.war /var/lib/tomcat7/webapps

Restart Tomcat.

sudo service tomcat7 restart

Login to subSonic by visiting http://server.example.org:8080/subsonic and login with the credentials admin and admin. Make sure you change the password straight away.

Right, that is it. You can stop here and start filling subSonic with your music.

subSonic clients

On the rare occasions that I listen to music via subSonic I use UltraSonic for Android and Clementine on my Arch Linux workstations.

References

Headless cloudprint server on Debian for MFC-7360N

martin@wimpress.com (Martin Wimpress) — Sat, 05 Jul 2014 12:00:00 +0000

I have a Brother MFC-7360N printer at home and there is also one at work. I wanted to to get Cloudprint working with Android devices rather than use the Android app Brother provide, which is great when it works but deeply frustrating (for my wife) when it doesn’t.

What I describe below is how to Cloudprint enable “Classic printers” using Debian Wheezy.

Install CUPS

Install CUPS and the Cloudprint requirements.

sudo apt-get install cups python-cups python-daemon python-pkg-resources

Install the MFC-7360N Drivers

I used the URL below to access the .deb files required.

http://support.brother.com/g/b/downloadlist.aspx?c=gb&lang=en&prod=mfc7360n_all&os=128

If you’re running a 64-bit Debian, then install ia32-libs first.

sudo apt-get install ia32-libs

Download and install the MFC-7360N drivers.

wget -c http://download.brother.com/welcome/dlf006237/mfc7360nlpr-2.1.0-1.i386.deb
wget -c http://download.brother.com/welcome/dlf006239/cupswrapperMFC7360N-2.0.4-2.i386.deb
sudo dpkg -i --force-all mfc7360nlpr-2.1.0-1.i386.deb
sudo dpkg -i --force-all cupswrapperMFC7360N-2.0.4-2.i386.deb

Configure CUPS

Edit the CUPS configuration file commonly located in /etc/cups/cupsd.conf and make the section that looks like this…

# Only listen for connections from the local machine.
Listen localhost:631
Listen /var/run/cups/cups.sock

…is changed to look like this:

# Listen on all interfaces
Port 631
Listen /var/run/cups/cups.sock

Modify the Apache specific directives to allow connections from everywhere as well. Find the follow section in /etc/cups/cupsd.conf:

<Location />
# Restrict access to the server...
 Order allow,deny
</Location>

# Restrict access to the admin pages...
<Location /admin>
 Order allow,deny
</Location>

# Restrict access to the configuration files...
<Location /admin/conf>
 AuthType Default
 Require user @SYSTEM
 Order allow,deny
</Location>

Add Allow All after each Order allow,deny so it looks like this:

<Location />
# Restrict access to the server...
 Order allow,deny
 Allow All
</Location>

# Restrict access to the admin pages...
<Location /admin>
 Order allow,deny
 Allow All
</Location>

# Restrict access to the configuration files...
<Location /admin/conf>
 AuthType Default
 Require user @SYSTEM
 Order allow,deny
 Allow All
</Location>

Add the MFC-7360N to CUPS

If your MFC-7360N is connected to your server via USB then you should be all set. Login to the CUPS administration interface on http://yourserver:631 and modify the MFC7360N printer (if one was created when the drivers where installed) then make sure you can print a test page via CUPS before proceeding.

Install Cloudprint and Cloudprint service

wget -c http://davesteele.github.io/cloudprint-service/deb/cloudprint_0.11-5.1_all.deb
wget -c http://davesteele.github.io/cloudprint-service/deb/cloudprint-service_0.11-5.1_all.deb
sudo dpkg -i cloudprint_0.11-5.1_all.deb
sudo dpkg -i cloudprint-service_0.11-5.1_all.deb

Authenticate

Google accounts with 2 step verification enabled need to use an application-specific password.

http://www.google.com/support/accounts/bin/static.py?page=guide.cs&guide=1056283&topic=1056286

Authenticate cloudprintd.

sudo service cloudprintd login

You should see something like this.

Accounts with 2 factor authentication require an application-specific password
Google username: you@example.org
Password:
Added Printer MFC7360N

Start the Cloudprint daemon.

sudo service cloudprintd start

If everything is working correctly you should see your printer the following page:

https://www.google.com/cloudprint#printers

Printing from mobile devices

Android

Add the Google Cloud Print app to Android devices and you’ll be able to configure your printer preferences and print from Android..

Chrome and Chromium

When printing from within Google Chrome and Chromium you can now select Cloudprint as the destination and choose your printer.

References

Humax Foxsat HDR Custom Firmware

martin@wimpress.com (Martin Wimpress) — Sun, 29 Jun 2014 12:00:00 +0000

I’ve had these notes kicking around for absolutely ages. I haven’t checked to see if this stuff is still accurate because during the last 12 months or so our viewing habits have changed and we almost exclusively watch streamed content now.

That said, my father-in-law gave us a Humax Foxsat HDR Freesat digital recorder as a thank you for some work I did for him. It turns out the Humax Foxsat HDR is quite hackable.

Hard Disk Upgrade

I contributed to the topic below. The Humax firmware only supports disks up to 1TB but the hackers method works for 2TB drives, possibly bigger but I haven’t tried. I’ve upgraded mine and my father-in-laws to 2TB without any problems.

http://www.avforums.com/forums/pvrs-vcrs/1336395-humax-foxsat-hdr-upgrade-hdd-2tb.html

Custom Firmware

These are the topics that discuss the custom firmware itself and includes the downloads. Once the custom firmware is installed and it’s advanced interface has been enabled you can enable several add-ons such as Samba, Mediatomb, ssh, ftp, etc.

Web Interface

This is the topic about the web interface. No download required it is bundled in the custom firmware above.

http://www.avforums.com/forums/freesat/1601205-web-interface-channel-editor-plug-foxsat-hdr.html

Channel Editor

The channel editor is installed and configured via the web interface and is one of my favourite add-ons. It also allows you to enable non-freesat channels in the normal guide.

http://www.avforums.com/forums/freesat/1601205-web-interface-channel-editor-plug-foxsat-hdr.html

Non Freesat channels

We get our broadcasts from Astra 2A / Astra 2B / Astra 2D / Eurobird 1 (28.2°E). Other free to air channels are available and KingOfSat lists them all. These channels can only be added via the Humax setup menus, but can then be presented in the normal EPG using the Channel Editor above.

http://en.kingofsat.net/pos-28.2E.php

Decrypt HD recordings

I never actually tried this, but what follows might be useful.

OTA updates

This is not so relevant now, since Humax haven’t released an OTA update for some time.

I have not yet found a way to prevent automatic over the air updates from being automatically applied. When an over the air update is applied the Humax still works, but the web interface and all the add-ons stop working. The can be solved by waiting for the custom firmware to be updated (which happen remarkably quickly) and then re-flashing the custom firmware. All the add-ons should start working again.

Integrating Dropbox photo syncing with Open Media Vault and Plex

martin@wimpress.com (Martin Wimpress) — Sat, 28 Jun 2014 12:00:00 +0000

This how-to was updated for Open Media Vault 2.x and 3.x on 22nd August 2016.

I’ve installed Open Media Vault on a HP ProLiant MicroServer G7 N54L and use it as media server for the house. OpenMediaVault (OMV) is a network attached storage (NAS) solution based on Debian Linux.

I use a free Dropbox account to sync photos from mine and my wife’s Android phones and wanted to automate to import of these photo upload into Plex, which is also running on Open Media Vault.

Installing Dropbox on Open Media Vault

I looked for a Dropbox Plugin for Open Media Vault and found this:

https://github.com/lordldx/openmediavault-dropbox

Sadly, at the time of writing, it is unfinished and I didn’t have the time to go and learn the Open Media Vault plugin API.

The Open Media Vault forum does include a Dropbox HOW-TO which is very similar to how I’ve run Dropbox on headless Linux servers in the past. So, I decided to adapt my existing notes to Open Media Vault.

Create a Dropbox share via the OMV WebUI.

Access Right Management -> Shared Folders

I gave my the name “Dropbox”. I know, very original.

Installing Dropbox on a headless server

Download and extract the latest Dropbox stable release.

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
chown -Rv ${USER}: ~/.dropbox-dist
sudo ln -s ~/.dropbox-dist/dropboxd /usr/local/bin/

Run dropboxd.

~/.dropbox-dist/dropboxd

You should see output like this:

This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?host_id=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx to link this machine.

Visit the URL, login with your Dropbox account and link the account. You should see the following.

Client successfully linked, Welcome Web!

dropboxd will now create a ~/Dropbox folder and start synchronizing. Stop dropboxd with CTRL+C.

Login to the OMV server as root and sym-link the Dropbox share you created earlier to the Dropbox directory in the root home directory.

mv ~/Dropbox ~/Dropbox-old
ln -s /media/<UUID>/Dropbox ~/Dropbox
rsync -a -W --progress ~/Dropbox-old/ ~/Dropbox/

init.d - Open Media Vault 2.x

If you are using Open Media Vault 2.x (based on Debian wheezy) the you’ll need to create an init script.

To run Dropbox as daemon with init.d. Create /etc/init.d/dropbox with the following content.

#!/bin/sh
### BEGIN INIT INFO
# Provides: dropbox
# Required-Start: $local_fs $remote_fs $network $syslog $named
# Required-Stop: $local_fs $remote_fs $network $syslog $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# X-Interactive: false
# Short-Description: dropbox service
### END INIT INFO

DROPBOX_USERS="user_a"
DAEMON=.dropbox-dist/dropboxd

start() {
 echo "Starting dropbox..."
 for dbuser in $DROPBOX_USERS; do
 HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
 if [ -x $DAEMON ]; then
 HOME="$HOMEDIR" start-stop-daemon -b -o -c $dbuser -S -u $dbuser -x $HOMEDIR/$DAEMON
 fi
 done
}

stop() {
 echo "Stopping dropbox..."
 for dbuser in $DROPBOX_USERS; do
 HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
 if [ -x $HOMEDIR/$DAEMON ]; then
 start-stop-daemon -o -c $dbuser -K -u $dbuser -x $HOMEDIR/$DAEMON
 fi
 done
}

status() {
 for dbuser in $DROPBOX_USERS; do
 dbpid=`pgrep -u $dbuser dropbox`
 if [ -z $dbpid ] ; then
 echo "dropboxd for USER $dbuser: not running."
 else
 echo "dropboxd for USER $dbuser: running (pid $dbpid)"
 fi
 done
}

case "$1" in
 start)
 start
 ;;
 stop)
 stop
 ;;
 restart|reload|force-reload)
 stop
 start
 ;;
 status)
 status
 ;;
 *)
 echo "Usage: /etc/init.d/dropbox {start|stop|reload|force-reload|restart|status}"
 exit 1
esac

exit 0

Enable the init.d script.

sudo chmod +x /etc/init.d/dropbox
sudo update-rc.d dropbox defaults

Starting and Stopping the Dropbox daemon

Use /etc/init.d/dropbox start to start and /etc/init.d/dropbox stop to stop.

systemd - OpenMediaVault 3.x

If you are using Open Media Vault 3.x (based on Debian jessie) then you’ll need to create a systemd unit. Create the systemd service file.

sudo nano /lib/systemd/system/dropbox.service

Add this:

[Unit]
Description=Dropbox
After=local-fs.target network.target

[Service]
Type=simple
WorkingDirectory=%h/.dropbox-dist
ExecStart=/usr/local/bin/dropboxd
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
User=%I

[Install]
WantedBy=multi-user.target

Enable the dropbox service for a given user.

sudo systemctl enable dropbox@user_a

Start the service.

sudo systemctl start dropbox@user_a

Dropbox client

It is recommended to download the official Dropbox client to configure Dropbox and get its status.

wget "http://www.dropbox.com/download?dl=packages/dropbox.py" -O dropbox
sudo chmod 755 dropbox
sudo mv dropbox /usr/local/bin/

You can check on Dropbox status by running the following.

dropbox status

For usage instructions run dropbox help.

Photo importing

So, the reason for doing all this is that I now have a Dropbox instance running on my home file server and everyday it runs a script, that I wrote, to automatically import new photos into a directory that Plex monitors. I’ll post details about my photo sorting script, Phort, at a later date.

References

http://www.dropboxwiki.com/Text_Based_Linux_Install

OpenMediaVault on Debian

martin@wimpress.com (Martin Wimpress) — Sun, 22 Jun 2014 12:00:00 +0000

At the time of writing OpenMediaVault 0.6 is pre-release. But it is possible to install OpenMediaVault on Debian Wheezy in order to get some testing done.

Install Debian Wheezy on your target VM or test server. Go with the defaults until the ‘Software selection’ dialogue. Make sure everything is unselected, like this:

[ ] Debian desktop environment
[ ] Web server
[ ] Print server
[ ] SQL database
[ ] DNS Server
[ ] File server
[ ] Mail server
[ ] SSH server
[ ] Laptop
[ ] Standard system utilities

After the install is complete, reboot and login to the new Debian system as root.

Update the repository sources and add the contrib and non-free repositories.

nano /etc/apt/sources.list

It should look something like this:

deb http://ftp.uk.debian.org/debian/ wheezy main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ wheezy main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free

# wheezy-updates, previously known as 'volatile'
deb http://ftp.uk.debian.org/debian/ wheezy-updates main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ wheezy-updates main contrib non-free

Now add the OpenMediaVault repository.

echo "deb http://packages.openmediavault.org/public kralizec main" > /etc/apt/sources.list.d/openmediavault.list

Update.

apt-get update

Install the OpenMediaVault repository key and Postfix.

apt-get install openmediavault-keyring postfix

When the ‘Postfix Configuration’ dialogue is displayed choose No configuration.

Update again and install OpenMediaVault.

apt-get update
apt-get install openmediavault

When the ‘Configuring mdadm’ dialogue is displayed enter none.
Do you want to start MD arrays automatically? YES
When the ‘ProFTPD configuration’ dialogue is displayed choose standalone.

Initialise OpenMediaVault and reboot.

omv-initsystem
reboot

After the reboot you should be able to connect to the OpenMediaVault WebUI and login as admin with the password of openmediavault. That’s it. Get testing.

Setting up BitSync on Debian

martin@wimpress.com (Martin Wimpress) — Sat, 21 Jun 2014 12:00:00 +0000

I’ve replaced Dropbox with BitTorrent Sync. In order to do this I’ve have btsync running on a VPS (2CPU, 2GB, 400GB), my home server and assorted Arch Linux workstations.

I had a couple of reasons for migrating away from Dropbox.

Dropbox was costing $100 per year.
Dropbox encryption model is weak and I have data security/privacy.

The VPS I am running BitTorrent Sync on costs $50 per year and provides four times the storage. I run btsync on a VPS so that there is always a server “in the cloud” that is available to sync with so that my setup emulates what Dropbox used to do.

All my servers are running Debian and this is how I install btsync on Debian.

sh -c "$(curl -fsSL http://debian.yeasoft.net/add-btsync-repository.sh)"
sudo apt-get install btsync

This is how I respond to the prompts:

* Do you want to define a default BitTorrent Sync instance? : YES
* BitTorrent Sync Daemon Credentials: yourusername
* BitTorrent Sync Daemon Group: yourusername
* Niceness of the BitTorrent Sync Daemon: 0
* On which portnumber should BitTorrent Sync listen? 0
* BitTorrent Sync Listen Port: 12345
* Do you want BitTorrent Sync to request port mapping via UPNP? NO
* Download Bandwith Limit: 0
* Upload Bandwith Limit: 0
* Web Interface Bind IP Address: 0.0.0.0
* Web Interface Listen Port: 8888
* The username for accessing the web interface: yourusername
* The password for accessing the web interface: yourpassword

As you’ll see, I don’t use UPNP on my VPS. I elect a specific port (not actually 12345 by the way) and open that port up with ufw. I also only allow access to the WebUI port from another server I own which reverse proxies via nginx.

btsync works really well, I have it syncing hundreds of thousands of files that amount to several hundred gigabytes of data. On my Arch Linux workstations I use the brilliant btsync-gui and BitTorrent Sync is also available for Android.

That said, I still use a free Dropbox account to sync photos from mine and my wife’s Android phones. I have a Dropbox instance running on my home file server and everyday it runs a script to automatically import these photos into Plex.

Such a shame, that at the time of writing, btsync is closed source :-( Maybe that will change but if it doesn’t syncthing may well be the answer when it has matured a little.

References

MATE Desktop on Debian Wheezy

martin@wimpress.com (Martin Wimpress) — Thu, 19 Jun 2014 22:00:00 +0000

I’m a member of the MATE Desktop team and until recently the majority of my involvement has been focused around Arch Linux.

However, I’m working on a MATE project that is based on a Debian derivative. MATE has recently been accepted into the Debian Backports repository for Wheezy, so I decided to do a “MATE from scratch” on Debian using an old netbook to get familiar with the MATE package naming differences between Arch Linux and Debian.

Install Debian

I installed Debian Wheezy from the netinst ISO to ensure the target install was as minimal as possible. I went with the defaults until the ‘Software selection’ dialogue, at this point unselect everything except “SSH server”. Like this:

[ ] Debian desktop environment
[ ] Web server
[ ] Print server
[ ] SQL database
[ ] DNS Server
[ ] File server
[ ] Mail server
[X] SSH server
[ ] Laptop
[ ] Standard system utilities

Debian ISO with Firmware

If you’re installing on hardware that requires additional firmware in order for it to work with Linux then use the netinst ISO that includes firmware.

http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/current/

Configure Debian

When the install is finished, reboot and configure Debian a little.

Repositories

You’ll need to install lsb-release for the following to work.

apt-get install lsb-release

This is what I put in /etc/apt/sources.list.

cat >/etc/apt/sources.list<<EOF
deb http://ftp.uk.debian.org/debian/ $(lsb_release -cs) main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ $(lsb_release -cs) main contrib non-free

deb http://security.debian.org/ $(lsb_release -cs)/updates main contrib non-free
deb-src http://security.debian.org/ $(lsb_release -cs)/updates main contrib non-free

# $(lsb_release -cs)-updates, previously known as 'volatile'
deb http://ftp.uk.debian.org/debian/ $(lsb_release -cs)-updates main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ $(lsb_release -cs)-updates main contrib non-free
EOF

Backports

MATE is only available in the wheezy-backports repository.

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.uk.debian.org/debian $(lsb_release -cs)-backports main contrib non-free
deb-src http://ftp.uk.debian.org/debian $(lsb_release -cs)-backports main contrib non-free
EOF

Update.

sudo apt-get update

All backports are deactivated by default (i.e. the packages are pinned to 100 by using ButAutomaticUpgrades: yes in the Release files. If you want to install something from backports run:

apt-get -t wheezy-backports install "package"

Install MATE Desktop

First install the LightDM display manager.

apt-get install accountsservice lightdm lightdm-gtk-greeter

Now for the MATE Desktop itself.

apt-get -t wheezy-backports install mate-desktop-environment-extras

NetworkManager

I typically use NetworkManager, so lets install that too.

apt-get install network-manager-gnome

Supplementary

Depending on your hardware you may require CPU frequency utilities or additional firmware.

apt-get install cpufreqd cpufrequtil firmware-linux firmware-linux-nonfree

And, that’s it! Reboot and you’ll see the LightDM greeter waiting for your login credentials.

References

Ubuntu MATE

martin@wimpress.com (Martin Wimpress) — Sun, 01 Jun 2014 00:00:00 +0000

A community developed Ubuntu based operating system that beautifully integrates the MATE desktop. Ubuntu MATE is a stable, easy-to-use operating system with a configurable desktop environment and is suitable for modern workstations, laptops, single board computers and older hardware alike.

Organisation: Ubuntu MATE Project
Date: June 2014 - date
Role: Project Lead

BIP IRC proxy

martin@wimpress.com (Martin Wimpress) — Wed, 16 Apr 2014 07:07:09 +0000

I switched from BIP to ZNC, and recommend you use ZNC instead!

BIP is an IRC proxy that maintains a persistent connection(s) to a list of IRC channels. You can then point your IRC client to BIP each time you log in and playback the conversations that took place while you were away.

I’ve found bBIP to be so useful that I now maintain BIP for Arch Linux, although I now run my BIP proxy on Debian because my new VPS provider doesn’t offer Arch Linux as an option.

Installing BIP

Installing BIP is simple for both Arch Linux and Debian.

Debian

I run BIP on Debian Wheezy with the backport repository enabled.

sudo apt-get -t wheezy-backports install bip
sudo sed -i 's/ENABLED=0/ENABLED=1/' /etc/default/bip

Arch Linux

pacman -S bip
systemctl enable bip

Create a user

The next thing to do is create a username and password and BIP provides it’s own utility for doing this called bipmkpw. Replace ‘username’ with whatever you want your BIP ‘username’ to be. This name has no relation to any IRC usernames so it can be anything.

bipmkpw username

Enter a password when prompted. The password will then be output as a hash. Make a note of both the hashed and un-hashed values somewhere, you will need them later.

Create a certificate

We don’t want the username and password being sent as clear-text, so we will create an SSL certificate for BIP to use.

openssl req -new -newkey rsa:4096 -nodes -x509 -keyout bip.pem -out bip.pem

Move the certificate to /var/lib/bip

sudo mv bip.pem /var/lib/bip

Change ownership and permissions of the certificate to the user bip which was created automatically when the package was installed.

sudo chown bip:bip /var/lib/bip/bip.pem
sudo chmod 600 /var/lib/bip/bip.pem

Configure BIP

Here is example configuration for BIP. Copy it to /etc/bip.conf, modify it accordingly and then change the ownership and permissions.

sudo chown bip:bip /etc/bip.conf
sudo chmod 640 /etc/bip.conf

Example configuration

# bip default config file.
# Thou shoult change thy password

ip = "0.0.0.0";

# To connect a client to bip, try the port below, and
# be sure to set the password to the value
# specified in the network you want to connect to.
port = 7778;

# If you set this to true, you'll only be able to connect to bip
# with a SSL capable IRC client. Be sure to generate a certificate
# for bip with 'make cert'
client_side_ssl = true;

log_level = 3;

pid_file="/var/run/bip/bip.pid";

# This is where logs go. Channel and private messages will use that
# configuration value as a prefix, and then log_format to determine
# full log filename.
log_root = "/var/log/bip/";

# Log format allows you to make log filenames depend on the log line's
# attributes. Here's a list :
# %u -> user name
# %n -> network name
# %Y -> 4 digit year
# %m -> 2 digit month
# %d -> 2 digit day
# %c -> destination (#chan, privates, ...)
#log_format = "%n/%Y-%m/%c.%d.log";

# Sets the frequency (in seconds) of log syncing (real write to kernel)
#log_sync_interval = 5;

# Makes bip send the log of each channel and privates while
# you were not connected to the proxy upon connection.
backlog = true; # enable backlog
backlog_lines = 0; # number of lines in backlog, 0 means no limit
backlog_always = false; # backlog even lines already backlogged

# If blreset_on_talk talking on an irc network has the same effect of issuing
# /bip blreset, meaning that stuffed logged before the command won't be read
# back on backlog
blreset_on_talk = true;

# Network definition, a name and server info
network {
 name = "freenode";
 server { host = "chat.freenode.net"; port = 6667; };
};

network {
 name = "blitzed";
 server { host = "irc.blitzed.org"; port = 6667; };
};

# Configuration example with one user who connects to two irc networks
# To use the multi-server feature:
# - define the connections
# - chose and setup a different login for each connection
# on your irc client:
# - Use the multi server feature of your client, the server being each time
# the server where bip is running. In your client setup server password to:
# username:password:connectionname
# - do not store the password in clear here, use the bipmkpw util to generate
# a hash

# User structure is grouping information for a given user
user {
 # The name in bip of the user
 # This is used by bip only
 name = "USERNAME; #BIP User account created with bipmkpw
 password = "00000000000000000000000000000000000000"; # the hash bipmkpw created
 ssl_check_mode = "none";

 # These will be the default for each connections
 default_nick = "NICKNAME"; #IRC Nick
 default_user = "IRCUSERNAME"; #IRC User
 default_realname = "REALNAME"; #IRC Real Name

 admin = true;
 backlog_msg_only = true; # When true,

 # A user can have mutiple connections to irc networks.
 # define a connection:

 connection {
 name = "freenode"; # used by bip only
 network = "freenode"; # which ircnet to connect to

 # these will be sent to the real IRC server
 user = "IRCUSERNAME";
 realname = "IRCREALNAME";
 password = "serverpassword"; #can be commented out if not needed

 # Some options:
 follow_nick = true;
 ignore_first_nick = false;
 #on_connect_send = "PRIVMSG NickServ :IDENTIFY nspassword";

 # Autojoined channels:
 channel { name = "#cat"; }; # Join #cat
 channel { name = "#dog"; backlog = false; }; # Join #dog but don't backlog it.
 channel { name = "#pig"; key = "01nk01nk"; }; # Join #pig that has a password.
 };

 connection {
 name = "blitzed"; # used by bip only
 network = "blitzed"; # which ircnet to connect to

 # these will be sent to the real IRC server
 user = "IRCUSERNAME";
 realname = "IRCREALNAME";
 password = "serverpassword"; #can be commented out if not needed

 # Some options:
 follow_nick = true;
 ignore_first_nick = false;
 #on_connect_send = "PRIVMSG NickServ :IDENTIFY nspassword";

 # Autojoined channels:
 channel { name = "#bar"; };
 channel { name = "#foo"; };
 };
};

If you require any clarification about what the configuration options do then man bip.conf is your friend.

Start BIP

Now that BIP is configured, it can be started.

Debian

sudo /etc/init.d/bip start

Arch Linux

sudo systemctl start bip

Client configuration

I use HexChat, but other IRC clients are available. I add a new Network to HexChat for each of the IRC networks I defined in /etc/bip.conf. The screen shot below shows how I configure a BIP network in HexChat.

Password format

The Password is the most important and confusing item. This is for BIP, not for any IRC network. Remember the unhashed password? That goes here but with a twist. The format for the password is:

bipusername:unhashedbippassword:bipnetwork

Bipnetwork? What is that? It is from the following section of /etc/bip.conf on the server?

network {
 name = "freenode";
 server { host = "chat.freenode.net"; port = 6667; };
};

A more practical example:

myuser:S3cr3tP@$$w0rd:freenode

Conclusion

And that’s it! We are now perpetually connected to IRC, can connect to BIP proxy from multiple devices in a completely transparent and seamless manner. Moreover, the logs for all channels are saved and automatically rotated on the server.

If you looking for an alternative to BIP, then try ZNC.

References

LXC on Arch Linux

martin@wimpress.com (Martin Wimpress) — Tue, 15 Apr 2014 02:44:11 +0000

At some point last year I was experimenting with Linux Containers (LXC) on Arch Linux. I never finished the blog post but somehow it was briefly published and then unplublished. I have no idea how accurate this blog post is but someone did see it and bookmarked it. They recently emailed me to ask where the blog has disappeared to, so here it is in all its unfinished glory.

Install LXC

sudo pacman -Syy --needed --noconfirm arch-install-scripts bridge-utils lxc netctl

netctl Bridge

The guest containers will connect to the LAN via a bridged network deviced.

sudo nano /etc/netctl/bridge

Add the following.

Description="Bridge"
Interface=br0
Connection=bridge
BindsToInterfaces=(eth0)
IP=dhcp
## sets forward delay time
FwdDelay=0
## sets max age of hello message
#MaxAge=10

Enable and start the bridge.

sudo netctl enable bridge
sudo netctl start bridge

Creating Containers

I’m only interested in running Arch Linux or Debian containers.

Container Configurations

Each container should have a matching configuration file, they look something like this.

lxc.arch = i686
lxc.utsname = myhostname
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.ipv4 = 0.0.0.0
lxc.network.name = eth0

lxc.arch Architecture for the container, valid options are x86, i686, x86_64, amd64.
lxc.utsman Container name, should also be used when naming the configuration file
lxc_network.type Type of network virtualization to be used for the container. The option veth defines a peer network device. It is created with one side assigned to the container and the other side is attached to a bridge by the lxc.network.link option.
lxc_network.flags Network actions. The value up in this case activates the network.
lxc.network.link Host network interface to be used for the container.
lxc.network.ipv4 IPv4 address assigned to the virtualized interface. Use the address 0.0.0.0 to make use of DHCP. Use lxc.network.ipv6 if you need IPv6 support.
lxc.network.name Dynamically allocated interface name. This option will rename the interface in the container.

More example files can be found in /usr/share/doc/lxc/examples/. Find details about all options via man lxc.conf.

Arch Linux

sudo lxc-create -t archlinux -n arch-01 -f ~/arch-01.conf -- --packages netctl

I am unable to get DHCP to work for a Arch Linux LXC container, therefore my dirty hack is to alway use a statis IP address in the netctl profile. There is also a bug (#35715) was helpful in narrowing down the problem, but wasn’t the solution in my case. Use /var/lib/lxc/CONTAIN_NAME/rootfs/etc/netctl/example/ethernet-static as a template.

sudo cp /var/lib/lxc/CONTAIN_NAME/rootfs/etc/netctl/example/ethernet-static /var/lib/lxc/CONTAIN_NAME/rootfs/etc/netctl/static

Modify /var/lib/lxc/CONTAIN_NAME/rootfs/etc/netctl/static accordingly. Now create a hook, with the same name as the netctl profile.

sudo nano /var/lib/lxc/CONTAIN_NAME/rootfs/etc/netctl/hooks/static

Add the following.

#!/usr/bin/env bash

if [[ $(systemd-detect-virt) != none ]]; then
 BindsToInterfaces=()
 ForceConnect=yes
fi

Start the container and enable the netctl profile.

netctl enable static
netctl start static

Debian Containers.

Install debobootstrap and dpkg so that Debian containers can be created.

packer -S --noedit dpkg debootstrap

Squeeze

Create a Debian container, squeeze is the default.

sudo lxc-create -t debian -n squeeze-01 -f ~/squeeze-01.conf

Change the root password.

chroot /var/lib/lxc/squeeze/rootfs/ passwd

Wheezy

Much the same as the Squeeze exaple above but use the following template.

https://github.com/simonvanderveldt/lxc-debian-wheezy-template

Using containers

Start a container

sudo lxc-start -d -n CONTAINER_NAME

Connect to the container and log in:

sudo lxc-console -n CONTAINER_NAME

To halt a container cleanly by the containers initv-system:

sudo lxc-halt -n CONTAINER_NAME

Stop and remove your container always with the two steps:

sudo lxc-stop -n CONTAINER_NAME
sudo lxc-destroy -n CONTAINER_NAME

References

Installing Nikola on Debian

martin@wimpress.com (Martin Wimpress) — Mon, 31 Mar 2014 16:19:43 +0000

Nikola is a static site and blog generator written in Python that I’ve been using for a good while now. This blog post describes how to install Nikola on Debian. Now, this may look like a long winded way to install Nikola, given that Debian .deb package exist, but in my opinion it is the correct way to install Nikola on Debian.

Installing Python

First you’ll need Python and virtualenvwrapper

sudo apt-get install libpython2.7 python2.7 python2.7-dev python2.7-minimal

Remove any apt installed Python packages that we are about to replace. The versions of these packages in the Debian repositories soon get stale.

sudo apt-get purge python-setuptools python-virtualenv python-pip python-profiler

Install pip.

wget https://bootstrap.pypa.io/get-pip.py
sudo python2.7 get-pip.py

Use pip to install virtualenv and virtualenvwrapper.

sudo pip install virtualenv --upgrade
sudo pip install virtualenvwrapper

The Snakepit

Create a “Snakepit” directory for storing all the virtualenvs.

mkdir ~/Snakepit

Add the following your ~/.bashrc to enable virtualenvwrapper.

export WORKON_HOME=${HOME}/Snakepit
if [ -f /usr/local/bin/virtualenvwrapper.sh ]; then
 source /usr/local/bin/virtualenvwrapper.sh
elif [ -f /usr/bin/virtualenvwrapper.sh ]; then
 source /usr/bin/virtualenvwrapper.sh
fi

Create a virtualenv for Nikola

Open a new shell to ensure that the virtualenvwrapper configuration is active. The following will create a new virtualenv called nikola based on Python 2.7.

mkvirtualenv -p /usr/bin/python2.7 ~/Snakepit/nikola-640

Working on a virtualenv

To activate an existing virtualenv do the following.

workon nikola-640

You can switch to another virtualenv at any time, just use workon envname. Your shell prompt will change while a virtualenv is being worked on to indicate which virtualenv is currently active.

When you are finished working in a virtualenv you can deactivate it by simply executing deactivate.

Install Nikola requirements

Nikola is will be powered by Python 2.7 and some additional packages will be required.

sudo apt-get install python2.7-dev libfreetype6-dev libjpeg8-dev libxslt1-dev libxml2-dev libyaml-dev

What are these requirements for?

python2.7-dev provides the header files for Python 2.7 so that Python modules with C extensions can be built.

The following are required to build pillow, the Python imaging library.

libjpeg8-dev
libfreetype6-dev

The following are required to build lxml, a Python XML library.

libxml2-dev
libxslt1-dev

The following are required to build python-coveralls.

libyaml-dev

Install Nikola

Download Nikola.

mkdir -p ${VIRTUAL_ENV}/src
cd ${VIRTUAL_ENV}/src
wget https://github.com/getnikola/nikola/archive/v6.4.0.tar.gz -O nikola-640.tar.gz
tar zxvf nikola-640.tar.gz
cd nikola-6.4.0

Install the Nikola requirements.

pip install -r requirements-full.txt

If you intend to use the Nikola planetoid (Planet generator) plugin you’ll also need to following.

pip install peewee feedparser

Actually install nikola.

python setup.py install

Nikola is now installed. nikola help and the Nikola Handbook will assist you from here on.

Memory consumption of Linux desktop environments

martin@wimpress.com (Martin Wimpress) — Sun, 23 Mar 2014 13:30:27 +0000

Note Updated on June 3rd 2014 and now includes LXQt and Unity.

For the last 9 months or so I’ve spent my spare time working with the MATE Desktop Team. Every so often, via the various on-line MATE communities, the topic of how “light weight” MATE is when compared to other desktop environments crops up and quite often Xfce is suggested as a lighter alternative. After all MATE and Xfce both provide a traditional desktop environment based on GTK+ so this suggestion is sensible. But is Xfce actually “lighter” than MATE?

I’ve found MATE to be (subjectively) more responsive that Xfce and there have been two recent blog posts that indicate MATE has lower memory requirements than Xfce.

Given that I’m comfortably running MATE on the Raspberry Pi Model B (which has just 512MB RAM) I’ve been stating that MATE is well suited for use on resource constrained hardware and professional workstations alike. This is still true, but I’ve also said that MATE is lighter than Xfce and I might have to eat humble pie on that one.

The topic of measursing desktop environment resource use came up on the #archlinux-tu IRC channel recently and someone suggested using ps_mem.py to gather the memory usage data. ps_mem.py provides a far more robust mechanism for gathering memory usage data than I’ve seen in previous comparisons.

So the seed was planted, I created a bunch of VirtualBox guest machines and set to work comparing the memory requirements of all the Linux desktop environments I could.

Damn it, just tell me what the “lightest” desktop environment is!

OK, for those of you who just want the final answer, with none of the explanation, here it is:

Desktop Environment	Memory Used
Enlightenment 0.18.8	83.8 MiB
LXDE 0.5.5	87.0 MiB
Xfce 4.10.2	110.0 MiB
LXQt 0.7.0	113.0 MiB
MATE 1.8.1	123.0 MiB
Cinnamon 2.2.13	176.3 MiB
GNOME3 3.12.2	245.3 MiB
KDE 4.13.1	302.6 MiB
Unity 7.2.0.14	312.5 MiB

Bullshit! How did you come up with these numbers?

All the VirtualBox VMs are 32-bit with 768MB RAM and based on the same core Arch Linux installation. I achieved this using my ArchInstaller script which is designed for quickly installing reproducible Arch Linux setups.

Each VM differs only by the packages that are required for the given desktop environment. The desktop environments native display manager is also installed but if it doesn’t have one then lightdm was chosen. LXDE, Xfce, MATE, Cinnamon and GNOME all have gvfs-smb installed as this enables accessing Windows and Samba shares (a common requirement for home and office) in their respective file managers and the KDE install includes packages to provide equivalent functionality. You can see the specific desktop environment packages or package groups that were installed here:

https://github.com/flexiondotorg/ArchInstaller/tree/master/packages/desktop

Each VM was booted, logged in and any initial desktop environment configuration was completed choosing the default options if prompted. Then ps_mem was installed, the VM shut down and a snapshot made.

Each VM was then started, logged in via the display manager, the desktop environment was fully loaded and waited for disk activity to settle. Then ps -efH and ps_mem were executed via SSH and the results sent back to my workstation. When the process and memory collections were conducted there had been no desktop interaction and no applications had been launched.

Your numbers are wrong I can get xxx desktop to run in yyy less memory!

Well done, you probably can.

Each virtual machine has VirtualBox guest additions, OpenSSH, Network Manager, avahi-daemon, ntpd, rpc.statd, syslog-ng and various other bits and bobs installed and running. Some of these are not required or have lighter alternatives available.

So, while I freely accept that each desktop environment can be run in less memory, the results here are relative to a consistent base setup.

However, what is important to note is that I think the Cinnamon results are too low. Cinnamon is forked from GNOME3 and the Arch Linux package groups for Cinnamon only install the core Cinnamon packages but none of the GNOME3 applications or components that would be required to create a full desktop environment.

So comparing Cinnamon with the other desktops in this test is not a fair comparison. For example, GNOME3 and KDE default installs on Arch Linux include all the accessibility extensions and applications for sight or mobility impaired individuals where as Cinnamon does not. This is just one example of where I think the Cinnamon results are skewed.

The RAM is there to be used. Is lighter actually better?

No, and Yes.

I subscribe to the school of thought that RAM is there to be used. But;

I want to preserve as much free RAM for the applications I run, not for feature bloat in the desktop environment. I’m looking at you KDE.
I want a fully integrated desktop experience, but not one that is merely lighter because it lacks features. I’m looking at you LXDE.
I want a consistent user interface that any of my family could use, not one that favours style over substance. I’m looking at you Enlightenment.

Another take on lightness is that the more RAM used, the more code that needs executing. Therefore, higher CPU utilisation and degraded desktop performance on modest hardware. This could also translate into degraded battery performance.

This is why I choose MATE Desktop. It is a fully integrated desktop environment, that is responsive, feature full, has reasonable memory requirements and scales from single core armv6h CPU with 512MB RAM to multi core x86_64 CPU with 32GB RAM (for me at least).

Without the full stats it never happened. Prove it!

He is the full data capture from ps_mem.py for each desktop environment.

Enlightenment

Private + Shared = RAM used Program

172.0 KiB + 46.5 KiB = 218.5 KiB dbus-launch
316.0 KiB + 40.0 KiB = 356.0 KiB dhcpcd
336.0 KiB + 87.5 KiB = 423.5 KiB rpcbind
560.0 KiB + 37.0 KiB = 597.0 KiB crond
580.0 KiB + 54.0 KiB = 634.0 KiB systemd-logind
688.0 KiB + 67.5 KiB = 755.5 KiB systemd-udevd
480.0 KiB + 276.0 KiB = 756.0 KiB avahi-daemon (2)
700.0 KiB + 133.5 KiB = 833.5 KiB ntpd
768.0 KiB + 78.5 KiB = 846.5 KiB VBoxService
580.0 KiB + 267.0 KiB = 847.0 KiB tempget
544.0 KiB + 312.0 KiB = 856.0 KiB enlightenment_start
764.0 KiB + 94.0 KiB = 858.0 KiB rpc.statd
600.0 KiB + 280.5 KiB = 880.5 KiB at-spi-bus-launcher
624.0 KiB + 298.0 KiB = 922.0 KiB at-spi2-registryd
724.0 KiB + 309.5 KiB = 1.0 MiB accounts-daemon
784.0 KiB + 386.5 KiB = 1.1 MiB enlightenment_fm
952.0 KiB + 395.0 KiB = 1.3 MiB efreetd
 1.0 MiB + 517.0 KiB = 1.5 MiB dbus-daemon (3)
 5.3 MiB + -3781.0 KiB = 1.7 MiB udisksd
 1.2 MiB + 483.0 KiB = 1.7 MiB (sd-pam) (2)
 1.6 MiB + 234.0 KiB = 1.9 MiB syslog-ng
 1.1 MiB + 1.0 MiB = 2.1 MiB systemd (3)
 1.4 MiB + 814.5 KiB = 2.2 MiB lightdm (2)
 1.3 MiB + 1.1 MiB = 2.4 MiB sshd (2)
 2.6 MiB + 575.5 KiB = 3.2 MiB VBoxClient (4)
 2.4 MiB + 781.0 KiB = 3.2 MiB NetworkManager
 10.9 MiB + -7741.5 KiB = 3.3 MiB polkitd
 6.2 MiB + 68.5 KiB = 6.3 MiB systemd-journald
 11.3 MiB + -2300.0 KiB = 9.1 MiB nm-applet
 16.3 MiB + 426.0 KiB = 16.7 MiB Xorg
 19.9 MiB + 1.5 MiB = 21.4 MiB enlightenment
---------------------------------
 89.6 MiB
=================================

LXDE

 Private + Shared = RAM used Program

184.0 KiB + 45.0 KiB = 229.0 KiB dbus-launch
320.0 KiB + 36.0 KiB = 356.0 KiB dhcpcd
340.0 KiB + 83.0 KiB = 423.0 KiB rpcbind
360.0 KiB + 78.0 KiB = 438.0 KiB lxdm-binary
384.0 KiB + 93.5 KiB = 477.5 KiB lxsession
580.0 KiB + 50.0 KiB = 630.0 KiB systemd-logind
700.0 KiB + 55.0 KiB = 755.0 KiB systemd-udevd
464.0 KiB + 297.0 KiB = 761.0 KiB avahi-daemon (2)
 4.6 MiB + -3890.5 KiB = 821.5 KiB menu-cached
612.0 KiB + 213.0 KiB = 825.0 KiB at-spi-bus-launcher
500.0 KiB + 328.0 KiB = 828.0 KiB lxdm-session
768.0 KiB + 97.5 KiB = 865.5 KiB rpc.statd
632.0 KiB + 251.5 KiB = 883.5 KiB gvfsd
644.0 KiB + 244.5 KiB = 888.5 KiB at-spi2-registryd
776.0 KiB + 189.0 KiB = 965.0 KiB accounts-daemon
 4.8 MiB + -3888.5 KiB = 1.0 MiB gvfsd-fuse
884.0 KiB + 305.0 KiB = 1.2 MiB gvfsd-trash
 1.1 MiB + 322.0 KiB = 1.4 MiB udisksd
 1.1 MiB + 381.0 KiB = 1.5 MiB upowerd
 1.1 MiB + 410.0 KiB = 1.5 MiB gvfs-udisks2-volume-monitor
 1.0 MiB + 485.5 KiB = 1.5 MiB dbus-daemon (3)
 1.2 MiB + 507.0 KiB = 1.7 MiB (sd-pam) (2)
 1.6 MiB + 259.0 KiB = 1.9 MiB syslog-ng
 1.2 MiB + 991.5 KiB = 2.1 MiB systemd (3)
 1.3 MiB + 1.1 MiB = 2.4 MiB sshd (2)
 1.5 MiB + 983.0 KiB = 2.4 MiB lxpolkit
 6.3 MiB + -3414.0 KiB = 3.0 MiB NetworkManager
 3.3 MiB + 706.5 KiB = 4.0 MiB openbox
 4.4 MiB + 59.5 KiB = 4.4 MiB systemd-journald
 6.9 MiB + -1941.0 KiB = 5.0 MiB lxpanel
 12.9 MiB + -7745.0 KiB = 5.3 MiB polkitd
 3.6 MiB + 1.8 MiB = 5.4 MiB pcmanfm
 11.5 MiB + -3637.5 KiB = 8.0 MiB ntpd
 7.1 MiB + 1.8 MiB = 9.0 MiB nm-applet
 13.9 MiB + 604.5 KiB = 14.5 MiB Xorg
---------------------------------
 87.0 MiB
=================================

Xfce

 Private + Shared = RAM used Program

176.0 KiB + 32.0 KiB = 208.0 KiB dbus-launch
292.0 KiB + 26.5 KiB = 318.5 KiB gpg-agent
316.0 KiB + 32.0 KiB = 348.0 KiB dhcpcd
324.0 KiB + 81.0 KiB = 405.0 KiB rpcbind
488.0 KiB + 96.0 KiB = 584.0 KiB xfconfd
588.0 KiB + 47.0 KiB = 635.0 KiB systemd-logind
464.0 KiB + 260.0 KiB = 724.0 KiB avahi-daemon (2)
712.0 KiB + 49.0 KiB = 761.0 KiB systemd-udevd
608.0 KiB + 173.0 KiB = 781.0 KiB at-spi-bus-launcher
644.0 KiB + 169.5 KiB = 813.5 KiB at-spi2-registryd
768.0 KiB + 57.5 KiB = 825.5 KiB VBoxService
784.0 KiB + 55.5 KiB = 839.5 KiB sh
640.0 KiB + 218.5 KiB = 858.5 KiB gvfsd
764.0 KiB + 94.5 KiB = 858.5 KiB rpc.statd
760.0 KiB + 160.0 KiB = 920.0 KiB accounts-daemon
872.0 KiB + 174.0 KiB = 1.0 MiB gvfsd-fuse
 4.8 MiB + -3831.0 KiB = 1.1 MiB gvfsd-trash
 1.1 MiB + 311.0 KiB = 1.4 MiB upowerd
 1.1 MiB + 282.0 KiB = 1.4 MiB tumblerd
 1.1 MiB + 289.0 KiB = 1.4 MiB udisksd
 1.1 MiB + 369.0 KiB = 1.4 MiB gvfs-udisks2-volume-monitor
 1.1 MiB + 353.0 KiB = 1.5 MiB xfce4-notifyd
 1.2 MiB + 515.0 KiB = 1.7 MiB (sd-pam) (2)
 1.3 MiB + 483.5 KiB = 1.8 MiB dbus-daemon (3)
 1.6 MiB + 248.5 KiB = 1.9 MiB syslog-ng
 1.5 MiB + 465.0 KiB = 1.9 MiB Thunar
 5.4 MiB + -3457.5 KiB = 2.0 MiB lightdm (2)
 1.1 MiB + 992.5 KiB = 2.1 MiB systemd (3)
 1.4 MiB + 695.5 KiB = 2.1 MiB panel-6-systray
 1.6 MiB + 651.0 KiB = 2.3 MiB xfce4-session
 1.3 MiB + 1.1 MiB = 2.3 MiB sshd (2)
 1.9 MiB + 525.0 KiB = 2.4 MiB xfsettingsd
 1.6 MiB + 903.0 KiB = 2.5 MiB panel-2-actions
 6.3 MiB + -3505.0 KiB = 2.9 MiB NetworkManager
 2.6 MiB + 442.5 KiB = 3.0 MiB VBoxClient (4)
 2.6 MiB + 624.5 KiB = 3.2 MiB xfce4-power-manager (2)
 2.1 MiB + 1.1 MiB = 3.2 MiB xfwm4
 3.1 MiB + 1.3 MiB = 4.4 MiB xfce4-panel
 5.0 MiB + 61.5 KiB = 5.0 MiB systemd-journald
 12.9 MiB + -7827.0 KiB = 5.3 MiB polkitd
 3.8 MiB + 1.6 MiB = 5.4 MiB xfdesktop
 6.6 MiB + 1.3 MiB = 7.8 MiB nm-applet
 11.5 MiB + -3643.5 KiB = 7.9 MiB ntpd
 23.0 MiB + -3258.0 KiB = 19.8 MiB Xorg
---------------------------------
 110.0 MiB
=================================

LXQt

 Private + Shared = RAM used Program

176.0 KiB + 35.0 KiB = 211.0 KiB dbus-launch
320.0 KiB + 35.0 KiB = 355.0 KiB dhcpcd
324.0 KiB + 83.0 KiB = 407.0 KiB rpcbind
612.0 KiB + 51.0 KiB = 663.0 KiB systemd-logind
460.0 KiB + 267.0 KiB = 727.0 KiB avahi-daemon (2)
676.0 KiB + 53.0 KiB = 729.0 KiB systemd-udevd
580.0 KiB + 179.0 KiB = 759.0 KiB menu-cached (2)
768.0 KiB + 63.5 KiB = 831.5 KiB VBoxService
604.0 KiB + 247.0 KiB = 851.0 KiB at-spi-bus-launcher
768.0 KiB + 96.5 KiB = 864.5 KiB rpc.statd
648.0 KiB + 231.5 KiB = 879.5 KiB at-spi2-registryd
4.7 MiB + -3856.0 KiB = 976.0 KiB accounts-daemon
908.0 KiB + 396.0 KiB = 1.3 MiB lxqt-globalkeysd
1.1 MiB + 425.0 KiB = 1.5 MiB upowerd
1.1 MiB + 484.5 KiB = 1.6 MiB dbus-daemon (3)
1.2 MiB + 551.0 KiB = 1.7 MiB (sd-pam) (2)
1.6 MiB + 248.0 KiB = 1.9 MiB syslog-ng
1.1 MiB + 998.5 KiB = 2.1 MiB systemd (3)
1.3 MiB + 1.0 MiB = 2.3 MiB sshd (2)
1.5 MiB + 933.5 KiB = 2.4 MiB lxqt-policykit-agent
1.6 MiB + 901.0 KiB = 2.4 MiB lxqt-session
1.7 MiB + 837.0 KiB = 2.5 MiB sddm
2.6 MiB + 456.5 KiB = 3.0 MiB VBoxClient (4)
6.3 MiB + -3350.0 KiB = 3.1 MiB NetworkManager
2.4 MiB + 1.5 MiB = 3.8 MiB lxqt-powermanagement
2.6 MiB + 1.5 MiB = 4.1 MiB lxqt-runner
3.4 MiB + 881.0 KiB = 4.3 MiB openbox
2.7 MiB + 1.8 MiB = 4.5 MiB lxqt-notificationd
4.7 MiB + 59.5 KiB = 4.7 MiB systemd-journald
12.5 MiB + -7812.0 KiB = 4.9 MiB polkitd
3.8 MiB + 2.2 MiB = 6.0 MiB lxqt-panel
11.5 MiB + -3644.5 KiB = 7.9 MiB ntpd
11.2 MiB + -2581.0 KiB = 8.7 MiB nm-applet
12.2 MiB + 366.0 KiB = 12.6 MiB pcmanfm-qt
18.2 MiB + -644.0 KiB = 17.6 MiB Xorg
---------------------------------
 113.0 MiB
=================================

MATE

 Private + Shared = RAM used Program

248.0 KiB + 57.0 KiB = 305.0 KiB rtkit-daemon
316.0 KiB + 31.0 KiB = 347.0 KiB dhcpcd
344.0 KiB + 81.0 KiB = 425.0 KiB rpcbind
388.0 KiB + 80.5 KiB = 468.5 KiB dbus-launch (2)
 4.4 MiB + -4003.0 KiB = 525.0 KiB dconf-service
588.0 KiB + 45.0 KiB = 633.0 KiB systemd-logind
552.0 KiB + 117.0 KiB = 669.0 KiB gconfd-2
456.0 KiB + 259.0 KiB = 715.0 KiB avahi-daemon (2)
548.0 KiB + 189.0 KiB = 737.0 KiB gconf-helper
692.0 KiB + 47.0 KiB = 739.0 KiB systemd-udevd
592.0 KiB + 150.5 KiB = 742.5 KiB at-spi-bus-launcher
660.0 KiB + 179.5 KiB = 839.5 KiB at-spi2-registryd
644.0 KiB + 199.5 KiB = 843.5 KiB gvfsd
768.0 KiB + 92.5 KiB = 860.5 KiB rpc.statd
 4.7 MiB + -3955.0 KiB = 893.0 KiB accounts-daemon
 4.8 MiB + -3943.5 KiB = 1.0 MiB gvfsd-fuse
 4.8 MiB + -3854.0 KiB = 1.1 MiB gvfsd-trash
 5.0 MiB + -3789.0 KiB = 1.3 MiB upowerd
 5.1 MiB + -3817.0 KiB = 1.4 MiB udisksd
 1.1 MiB + 337.0 KiB = 1.4 MiB gvfs-udisks2-volume-monitor
 1.6 MiB + 249.5 KiB = 1.9 MiB syslog-ng
 1.6 MiB + 429.0 KiB = 2.0 MiB polkit-mate-authentication-agent-1
 5.4 MiB + -3485.5 KiB = 2.0 MiB lightdm (2)
 1.7 MiB + 510.5 KiB = 2.2 MiB dbus-daemon (4)
 1.3 MiB + 1.0 MiB = 2.3 MiB sshd (2)
 1.4 MiB + 1.0 MiB = 2.4 MiB systemd (4)
 1.8 MiB + 679.5 KiB = 2.4 MiB (sd-pam) (3)
 1.9 MiB + 570.5 KiB = 2.5 MiB mate-screensaver
 2.0 MiB + 536.0 KiB = 2.5 MiB mate-session
 1.9 MiB + 679.5 KiB = 2.6 MiB notification-area-applet
 2.1 MiB + 703.0 KiB = 2.8 MiB mate-power-manager
 2.2 MiB + 596.0 KiB = 2.8 MiB NetworkManager
 2.7 MiB + 686.5 KiB = 3.4 MiB marco
 2.6 MiB + 937.5 KiB = 3.5 MiB wnck-applet
 3.6 MiB + 309.5 KiB = 3.9 MiB pulseaudio
 2.7 MiB + 1.2 MiB = 3.9 MiB mate-volume-control-applet
 3.0 MiB + 1.0 MiB = 4.0 MiB clock-applet
 7.6 MiB + -2931.0 KiB = 4.7 MiB mate-settings-daemon
 7.7 MiB + -2790.0 KiB = 4.9 MiB mate-panel
 5.0 MiB + 60.5 KiB = 5.1 MiB systemd-journald
 13.0 MiB + -7854.0 KiB = 5.3 MiB polkitd
 10.2 MiB + -2592.0 KiB = 7.7 MiB caja
 11.5 MiB + -3653.5 KiB = 7.9 MiB ntpd
 7.5 MiB + 1.1 MiB = 8.6 MiB nm-applet
 14.9 MiB + 1.2 MiB = 16.1 MiB Xorg
---------------------------------
 123.0 MiB
=================================

Cinnamon

 Private + Shared = RAM used Program

248.0 KiB + 56.0 KiB = 304.0 KiB rtkit-daemon
316.0 KiB + 31.0 KiB = 347.0 KiB dhcpcd
340.0 KiB + 82.0 KiB = 422.0 KiB rpcbind
 4.4 MiB + -3995.0 KiB = 469.0 KiB dconf-service
384.0 KiB + 88.5 KiB = 472.5 KiB dbus-launch (2)
576.0 KiB + 44.0 KiB = 620.0 KiB systemd-logind
556.0 KiB + 115.0 KiB = 671.0 KiB gconfd-2
452.0 KiB + 258.0 KiB = 710.0 KiB avahi-daemon (2)
544.0 KiB + 185.0 KiB = 729.0 KiB gconf-helper
596.0 KiB + 174.5 KiB = 770.5 KiB at-spi-bus-launcher
656.0 KiB + 170.5 KiB = 826.5 KiB at-spi2-registryd
800.0 KiB + 47.0 KiB = 847.0 KiB systemd-udevd
640.0 KiB + 208.5 KiB = 848.5 KiB gvfsd
768.0 KiB + 88.5 KiB = 856.5 KiB rpc.statd
 4.7 MiB + -3946.0 KiB = 910.0 KiB accounts-daemon
 4.8 MiB + -3938.5 KiB = 1.0 MiB gvfsd-fuse
 4.8 MiB + -3847.0 KiB = 1.1 MiB gvfsd-trash
 5.0 MiB + -3818.0 KiB = 1.3 MiB upowerd
 5.1 MiB + -3819.0 KiB = 1.4 MiB udisksd
 1.1 MiB + 340.0 KiB = 1.4 MiB gvfs-udisks2-volume-monitor
 1.1 MiB + 303.0 KiB = 1.4 MiB cupsd
 1.3 MiB + 394.0 KiB = 1.7 MiB csd-printer
 1.6 MiB + 198.5 KiB = 1.8 MiB syslog-ng
 1.4 MiB + 599.5 KiB = 2.0 MiB lightdm (2)
 1.6 MiB + 513.5 KiB = 2.1 MiB dbus-daemon (4)
 1.3 MiB + 979.0 KiB = 2.2 MiB sshd (2)
 1.4 MiB + 1.0 MiB = 2.4 MiB systemd (4)
 1.9 MiB + 784.5 KiB = 2.6 MiB (sd-pam) (3)
 6.2 MiB + -3555.0 KiB = 2.7 MiB NetworkManager
 6.6 MiB + -3794.5 KiB = 2.9 MiB colord
 2.7 MiB + 713.5 KiB = 3.4 MiB polkit-gnome-authentication-agent-1
 2.8 MiB + 805.0 KiB = 3.6 MiB cinnamon-screensaver
 3.6 MiB + 341.5 KiB = 3.9 MiB pulseaudio
 3.2 MiB + 826.5 KiB = 4.0 MiB cinnamon-session
 4.9 MiB + 56.5 KiB = 5.0 MiB systemd-journald
 13.2 MiB + -7890.0 KiB = 5.4 MiB polkitd
 3.9 MiB + 2.1 MiB = 6.0 MiB nm-applet
 5.5 MiB + 2.0 MiB = 7.5 MiB cinnamon-settings-daemon
 11.4 MiB + -3668.5 KiB = 7.9 MiB ntpd
 8.2 MiB + 1.1 MiB = 9.3 MiB cinnamon-launch
 7.7 MiB + 2.0 MiB = 9.8 MiB nemo
 21.2 MiB + -527.5 KiB = 20.7 MiB Xorg
 85.9 MiB + -34668.5 KiB = 52.1 MiB cinnamon
---------------------------------
 176.3 MiB
=================================

GNOME3

 Private + Shared = RAM used Program

180.0 KiB + 34.0 KiB = 214.0 KiB dbus-launch
276.0 KiB + 14.0 KiB = 290.0 KiB ssh-agent
248.0 KiB + 51.0 KiB = 299.0 KiB rtkit-daemon
312.0 KiB + 28.0 KiB = 340.0 KiB dhcpcd
324.0 KiB + 21.5 KiB = 345.5 KiB systemd-hostnamed
328.0 KiB + 20.0 KiB = 348.0 KiB systemd-localed
324.0 KiB + 80.0 KiB = 404.0 KiB rpcbind
580.0 KiB + 52.5 KiB = 632.5 KiB bluetoothd
604.0 KiB + 40.0 KiB = 644.0 KiB systemd-logind
556.0 KiB + 109.0 KiB = 665.0 KiB gconfd-2
452.0 KiB + 252.0 KiB = 704.0 KiB avahi-daemon (2)
680.0 KiB + 44.0 KiB = 724.0 KiB systemd-udevd
548.0 KiB + 182.0 KiB = 730.0 KiB gconf-helper
 4.6 MiB + -3950.5 KiB = 765.5 KiB at-spi2-registryd
596.0 KiB + 185.0 KiB = 781.0 KiB at-spi-bus-launcher
768.0 KiB + 50.5 KiB = 818.5 KiB VBoxService
696.0 KiB + 146.5 KiB = 842.5 KiB gvfsd
768.0 KiB + 86.5 KiB = 854.5 KiB rpc.statd
 4.8 MiB + -3960.0 KiB = 960.0 KiB accounts-daemon
852.0 KiB + 137.5 KiB = 989.5 KiB gvfsd-fuse
792.0 KiB + 267.0 KiB = 1.0 MiB zeitgeist-daemon
 5.1 MiB + -3910.5 KiB = 1.3 MiB gdm
 5.0 MiB + -3853.0 KiB = 1.3 MiB upowerd
 1.0 MiB + 291.0 KiB = 1.3 MiB gvfs-udisks2-volume-monitor
 5.1 MiB + -3877.0 KiB = 1.3 MiB udisksd
 1.1 MiB + 286.0 KiB = 1.4 MiB cupsd
 1.4 MiB + 133.0 KiB = 1.5 MiB gnome-keyring-daemon
 1.1 MiB + 448.0 KiB = 1.5 MiB gdm-session-worker
 1.3 MiB + 359.0 KiB = 1.7 MiB gsd-printer
 1.2 MiB + 509.0 KiB = 1.7 MiB (sd-pam) (2)
 1.6 MiB + 192.5 KiB = 1.8 MiB syslog-ng
 1.3 MiB + 648.0 KiB = 1.9 MiB mission-control-5
 5.6 MiB + -3676.5 KiB = 2.0 MiB gnome-session
 1.1 MiB + 984.5 KiB = 2.1 MiB systemd (3)
 1.5 MiB + 573.5 KiB = 2.1 MiB zeitgeist-datahub
 1.3 MiB + 953.0 KiB = 2.2 MiB sshd (2)
 6.1 MiB + -3700.5 KiB = 2.5 MiB colord
 6.1 MiB + -3656.0 KiB = 2.5 MiB NetworkManager
 2.2 MiB + 466.0 KiB = 2.6 MiB dbus-daemon (3)
 2.0 MiB + 728.5 KiB = 2.7 MiB gnome-shell-calendar-server
 2.6 MiB + 491.5 KiB = 3.1 MiB VBoxClient (4)
 2.5 MiB + 1.0 MiB = 3.5 MiB evolution-source-registry
 6.5 MiB + -2914.5 KiB = 3.6 MiB tracker-extract
 3.5 MiB + 338.5 KiB = 3.9 MiB pulseaudio
 6.7 MiB + -2828.5 KiB = 3.9 MiB tracker-miner-fs
 3.4 MiB + 2.1 MiB = 5.5 MiB goa-daemon
 13.3 MiB + -7973.0 KiB = 5.5 MiB polkitd
 4.9 MiB + 728.0 KiB = 5.7 MiB tracker-store
 6.1 MiB + 51.5 KiB = 6.2 MiB systemd-journald
 4.2 MiB + 2.1 MiB = 6.3 MiB nm-applet
 11.4 MiB + -3668.5 KiB = 7.9 MiB ntpd
 10.1 MiB + -1729.0 KiB = 8.4 MiB gnome-settings-daemon
 8.1 MiB + 1.7 MiB = 9.8 MiB Xorg
 10.7 MiB + -816.5 KiB = 9.9 MiB evolution-alarm-notify
 24.6 MiB + 1.1 MiB = 25.7 MiB evolution-calendar-factory
143.2 MiB + -56658.5 KiB = 87.9 MiB gnome-shell
---------------------------------
 245.3 MiB
=================================

KDE

 Private + Shared = RAM used Program

 72.0 KiB + 8.0 KiB = 80.0 KiB start_kdeinit
 80.0 KiB + 13.5 KiB = 93.5 KiB kwrapper4
128.0 KiB + 23.0 KiB = 151.0 KiB agetty
176.0 KiB + 28.0 KiB = 204.0 KiB dbus-launch
292.0 KiB + 28.5 KiB = 320.5 KiB gpg-agent
320.0 KiB + 28.0 KiB = 348.0 KiB dhcpcd
272.0 KiB + 84.0 KiB = 356.0 KiB cat (4)
340.0 KiB + 79.0 KiB = 419.0 KiB rpcbind
604.0 KiB + 39.0 KiB = 643.0 KiB systemd-logind
464.0 KiB + 247.0 KiB = 711.0 KiB avahi-daemon (2)
768.0 KiB + 87.5 KiB = 855.5 KiB rpc.statd
852.0 KiB + 51.5 KiB = 903.5 KiB startkde
352.0 KiB + 669.0 KiB = 1.0 MiB systemd-udevd (2)
656.0 KiB + 524.5 KiB = 1.2 MiB kdm (2)
 1.1 MiB + 420.0 KiB = 1.5 MiB upowerd
852.0 KiB + 780.0 KiB = 1.6 MiB klauncher
 1.3 MiB + 346.0 KiB = 1.7 MiB udisksd
 1.5 MiB + 256.5 KiB = 1.8 MiB akonadi_control
 1.3 MiB + 529.0 KiB = 1.8 MiB (sd-pam) (2)
 1.6 MiB + 201.5 KiB = 1.8 MiB syslog-ng
 1.5 MiB + 409.5 KiB = 1.9 MiB dbus-daemon (2)
 1.2 MiB + 971.5 KiB = 2.1 MiB systemd (3)
656.0 KiB + 1.5 MiB = 2.2 MiB kdeinit4
 1.3 MiB + 997.0 KiB = 2.3 MiB sshd (2)
 1.3 MiB + 1.6 MiB = 2.9 MiB kio_trash (2)
 2.1 MiB + 1.1 MiB = 3.2 MiB klipper
 6.9 MiB + -3587.0 KiB = 3.4 MiB NetworkManager
 2.4 MiB + 1.0 MiB = 3.5 MiB ksmserver
 3.2 MiB + 594.5 KiB = 3.8 MiB kuiserver
 3.2 MiB + 952.5 KiB = 4.1 MiB kglobalaccel
 3.4 MiB + 829.5 KiB = 4.2 MiB akonadi_migration_agent
 3.4 MiB + 837.5 KiB = 4.3 MiB polkit-kde-authentication-agent-1
 3.8 MiB + 716.5 KiB = 4.5 MiB knotify4
 4.4 MiB + 49.5 KiB = 4.5 MiB systemd-journald
 3.8 MiB + 891.0 KiB = 4.7 MiB baloo_file
 3.8 MiB + 975.0 KiB = 4.7 MiB akonadi_maildispatcher_agent
 3.8 MiB + 983.0 KiB = 4.7 MiB akonadi_baloo_indexer
 4.1 MiB + 1.3 MiB = 5.4 MiB akonadi_newmailnotifier_agent
 4.3 MiB + 1.1 MiB = 5.4 MiB korgac
 13.2 MiB + -7804.0 KiB = 5.6 MiB polkitd
 5.5 MiB + 1.7 MiB = 7.1 MiB akonadi_notes_agent
 11.1 MiB + -3575.0 KiB = 7.6 MiB kactivitymanagerd
 5.6 MiB + 2.1 MiB = 7.7 MiB akonadi_sendlater_agent
 11.4 MiB + -3697.5 KiB = 7.8 MiB ntpd
 7.2 MiB + 777.5 KiB = 8.0 MiB akonadiserver
 6.0 MiB + 2.9 MiB = 8.9 MiB akonadi_archivemail_agent
 6.3 MiB + 2.6 MiB = 8.9 MiB kmix
 6.1 MiB + 2.9 MiB = 9.0 MiB akonadi_mailfilter_agent
 6.9 MiB + 2.4 MiB = 9.3 MiB kded4
 9.1 MiB + 2.7 MiB = 11.7 MiB akonadi_agent_launcher (3)
 13.8 MiB + -1069.5 KiB = 12.8 MiB kwin
 13.2 MiB + 3.2 MiB = 16.5 MiB krunner
 68.8 MiB + -49024.0 KiB = 21.0 MiB mysqld
 30.3 MiB + -2270.0 KiB = 28.1 MiB Xorg
 36.9 MiB + 6.8 MiB = 43.7 MiB plasma-desktop
---------------------------------
 302.6 MiB
=================================

Unity

 Private + Shared = RAM used Program

 92.0 KiB + 14.0 KiB = 106.0 KiB cat
128.0 KiB + 21.0 KiB = 149.0 KiB agetty
180.0 KiB + 28.0 KiB = 208.0 KiB dbus-launch
252.0 KiB + 47.0 KiB = 299.0 KiB rtkit-daemon
312.0 KiB + 27.0 KiB = 339.0 KiB dhcpcd
324.0 KiB + 18.0 KiB = 342.0 KiB systemd-localed
336.0 KiB + 21.0 KiB = 357.0 KiB systemd-timedated
344.0 KiB + 79.0 KiB = 423.0 KiB rpcbind
412.0 KiB + 69.0 KiB = 481.0 KiB dconf-service
592.0 KiB + 39.0 KiB = 631.0 KiB systemd-logind
588.0 KiB + 92.5 KiB = 680.5 KiB indicator-messages-service
448.0 KiB + 245.0 KiB = 693.0 KiB avahi-daemon (2)
604.0 KiB + 89.5 KiB = 693.5 KiB indicator-bluetooth-service
548.0 KiB + 168.0 KiB = 716.0 KiB gconf-helper
 4.6 MiB + -3955.0 KiB = 725.0 KiB at-spi-bus-launcher
684.0 KiB + 42.0 KiB = 726.0 KiB systemd-udevd
640.0 KiB + 100.0 KiB = 740.0 KiB gconfd-2
636.0 KiB + 115.5 KiB = 751.5 KiB at-spi2-registryd
652.0 KiB + 144.5 KiB = 796.5 KiB indicator-power-service
648.0 KiB + 158.5 KiB = 806.5 KiB gvfsd
772.0 KiB + 45.5 KiB = 817.5 KiB VBoxService
768.0 KiB + 85.5 KiB = 853.5 KiB rpc.statd
796.0 KiB + 122.0 KiB = 918.0 KiB accounts-daemon
568.0 KiB + 391.5 KiB = 959.5 KiB (sd-pam)
 4.8 MiB + -3982.5 KiB = 969.5 KiB gvfsd-fuse
824.0 KiB + 253.0 KiB = 1.1 MiB dbus (2)
828.0 KiB + 271.0 KiB = 1.1 MiB zeitgeist-daemon
 4.9 MiB + -3904.0 KiB = 1.1 MiB gvfsd-trash
 1.0 MiB + 131.5 KiB = 1.2 MiB indicator-session-service
 5.0 MiB + -3883.0 KiB = 1.2 MiB upowerd
 1.1 MiB + 237.0 KiB = 1.3 MiB cupsd
 1.1 MiB + 279.0 KiB = 1.4 MiB gvfs-udisks2-volume-monitor
 5.1 MiB + -3857.0 KiB = 1.4 MiB udisksd
 1.1 MiB + 324.5 KiB = 1.4 MiB zeitgeist-fts
 5.1 MiB + -3833.0 KiB = 1.4 MiB indicator-application-service
 1.2 MiB + 382.0 KiB = 1.6 MiB indicator-sound-service
 1.6 MiB + 174.5 KiB = 1.8 MiB syslog-ng
 5.6 MiB + -3769.0 KiB = 1.9 MiB gnome-session
932.0 KiB + 1.0 MiB = 1.9 MiB systemd (2)
 1.4 MiB + 537.0 KiB = 2.0 MiB lightdm (2)
 1.4 MiB + 753.0 KiB = 2.1 MiB mission-control-5
 1.3 MiB + 936.0 KiB = 2.2 MiB sshd (2)
 1.8 MiB + 531.0 KiB = 2.3 MiB zeitgeist-datahub
 6.2 MiB + -3779.0 KiB = 2.5 MiB colord
 2.2 MiB + 458.5 KiB = 2.6 MiB dbus-daemon (3)
 6.2 MiB + -3682.0 KiB = 2.6 MiB NetworkManager
 2.3 MiB + 515.0 KiB = 2.8 MiB gnome-fallback-mount-helper
 2.3 MiB + 485.0 KiB = 2.8 MiB polkit-gnome-authentication-agent-1
 2.4 MiB + 514.0 KiB = 2.9 MiB notify-osd
 2.5 MiB + 417.5 KiB = 3.0 MiB VBoxClient (4)
 2.6 MiB + 547.5 KiB = 3.1 MiB indicator-keyboard-service
 2.7 MiB + 657.0 KiB = 3.3 MiB indicator-printers-service
 2.7 MiB + 854.0 KiB = 3.5 MiB telepathy-indicator
 2.5 MiB + 1.1 MiB = 3.6 MiB evolution-source-registry
 3.1 MiB + 577.0 KiB = 3.7 MiB bamfdaemon
 3.6 MiB + 306.5 KiB = 3.9 MiB pulseaudio
 3.0 MiB + 933.0 KiB = 3.9 MiB indicator-datetime-service
 3.9 MiB + 1.3 MiB = 5.3 MiB unity-panel-service
 13.2 MiB + -7982.0 KiB = 5.4 MiB polkitd
 4.4 MiB + 1.8 MiB = 6.1 MiB nm-applet
 6.1 MiB + 747.0 KiB = 6.9 MiB goa-daemon
 7.3 MiB + 50.5 KiB = 7.3 MiB systemd-journald
 11.4 MiB + -3690.5 KiB = 7.8 MiB ntpd
 5.3 MiB + 4.5 MiB = 9.8 MiB gnome-settings-daemon
 11.6 MiB + 1.0 MiB = 12.6 MiB nautilus
 24.7 MiB + 1.2 MiB = 25.8 MiB evolution-calendar-factory
 40.2 MiB + -351.0 KiB = 39.9 MiB Xorg
103.2 MiB + -4818.0 KiB = 98.5 MiB compiz
---------------------------------
 312.5 MiB
=================================

Final thoughts

On Arch Linux at least, Xfce has lower resource requirements than MATE. When I said different in the past I was wrong, unless you use openSUSE in which case I was probably right, maybe.

openSUSE

martin@wimpress.com (Martin Wimpress) — Sat, 01 Feb 2014 00:00:00 +0000

Mentored three students for the openSUSE 2014 Google Summer of Code. All three students successfully completed their projects for the MATE Desktop.

Organisation: openSUSE
Date: February 2014 - August 2014
Role: Student mentor

LINUX Unplugged

martin@wimpress.com (Martin Wimpress) — Wed, 01 Jan 2014 00:00:00 +0000

LINUX Unplugged is a weekly show, by Jupiter Broadcasting, that debates and discusses developments and goings on in the Linux community. I joined as a regular contributor in January 2014.

Organisation: Jupiter Broadcasting
Date: January 2014 - February 2021
Role: Talking Head

Arch Linux

martin@wimpress.com (Martin Wimpress) — Sun, 01 Dec 2013 00:00:00 +0000

Arch Linux is a Linux distribution that uses a rolling release model, such that a regular system update is all that is needed to obtain all the latest software. The design approach follows the KISS principle (“keep it simple, stupid”) and focuses on elegance, code correctness, minimalism and simplicity.

Organisation: Arch Linux
Date: December 2013 - December 2016
Role: Trusted user & Package maintainer

Linux Mint LMDE on Hybrid Disk Laptop

martin@wimpress.com (Martin Wimpress) — Wed, 28 Aug 2013 20:10:09 +0000

I have an old Thinkpad T43p that I am trying to extend the life of. So I recently fitted a cheap 60GB IDE Solid State Drive (SSD) and put a 320GB SATA Hard Disk Drive (HDD) in the Ultrabay. This is not a true hybrid disk, but the principles are similar. The root partition will go on the SDD (for performance) and the home partition will be located on the HDD (for capacity).

I’ve been running Arch Linux on the T43p and the SDD improves system responsiveness and boot time considerably, especially when using F2FS or btrfs (with LZO compression and SSD mount options) on the root filesystem.

I am also testing Linux Mint Debian Edition (LMDE) with the MATE Desktop desktop to determine if this is a suitable operating system for my family to use. It appears they find GNOME 3 confusing and would prefer a familiar desktop experience.

While testing LMDE 201203 I ran into a few issues, so I’ve decided to capture my notes here for future reference.

Fix the installer

In order to install LMDE using partitions on multiple drives you must use the ADVANCED USER install mode. However, the ADVANCED USER install mode has a bug that prevents the installer from completing, so that needs to be fixed first.

sudo nano /usr/lib/live-installer/frontend/gtk_interface.py

Find the following on line 1765.

self.wTree.get_widget("button_next").show()

After it add the following line, making sure the indentation is correct.

self.wTree.get_widget("button_next").set_sensitive(True)

Installing LMDE

Double click the Install Linux Mint icon on the desktop.
Select your Language and click Forward.
Select your Timezone and click Forward.
Select your Keyboard layout and click Forward.
Enter your User info and click Forward.
From the Hard drive window Select Manually mount partitions (ADVANCED USERS ONLY) and click Forward.

The Please make sure you wish to manually manage partitions window will appear. On my system the SSD is detected on /dev/sda and the HDD is detected on /dev/sdb. Start GParted and partition and format the drives as follows.

 /dev/sda1
 Size: 256MiB
 Create as: Primary Partition
 File system: ext4
 Label: boot

 /dev/sda2
 Size: 2048MiB (or the size you prefer)
 Create as: Primary Partition
 File system: linux-swap
 Label: swap

 /dev/sda3
 Size: Remainder
 Create as: Primary Partition
 File system: btrfs
 Label: root

 /dev/sdb1
 Size: All
 Create as: Primary Partition
 File system: ext4
 Label: home

Apply the changes and close GParted.

I use ext4 for the /boot partition because GRUB can’t currently boot from btrfs in LMDE. I use ext4 for /home because it offers the best performance on rotational drives on my Thinkpad T43p. I use btrfs on the /root partition because performs best (by some margin) on solid state drives in my Thinkpad T43p.

The filesystems need mounting under /target so the installer can install the OS. Open a Terminal and do the following.

sudo mkdir /target
sudo mount -t btrfs -o compress=lzo,ssd /dev/disk/by-label/root /target
sudo mkdir /target/{boot,home}
sudo mount -t ext4 /dev/disk/by-label/boot /target/boot
sudo mount -t ext4 /dev/disk/by-label/home /target/home

Return to the Linux Mint Debian Installer.

Click Forward.
From the Advanced options check Install GRUB and select /dev/sda. Click Forward.
Confirm the Summary is correct and click Install.

Time for a cup of tea while the install runs. A pop-up, titled Installation Paused, will appear. Click OK.

Create `/target/etc/fstab`

Do as the installer says and create /target/etc/fstab. Open a Terminal.

sudo nano /target/etc/fstab

The following fstab works for my T43p.

# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
LABEL=root / btrfs defaults,noatime,compress=lzo,ssd 0 1
LABEL=boot /boot ext4 defaults,noatime 0 2
LABEL=home /home ext4 defaults,relatime 0 2
LABEL=swap none swap sw 0 0

Upgrade MATE 1.6 and remove legacy MATE 1.4 packages

MATE 1.6 has been released for LMDE 201203 so it is a good idea to upgrade and remove legacy packages before the first boot to ensure a clean configuration. Open a Terminal.

sudo chroot /target
apt-get update
apt-get install apt-show-versions
apt-get dist-upgrade

Some MATE 1.4 packages will be left behind that are no longer required. The following can help identify them.

apt-show-versions | grep 1\.4\.[0-9]\-[0-9]\+wheezy | cut -f1 -d' '

Purge the old MATE 1.4 packages.

PKGS=`apt-show-versions | grep 1\.4\.[0-9]\-[0-9]\+wheezy | cut -f1 -d' '`
apt-get purge ${PKGS}

Exit the chroot.

exit

Finish the install

Return to the Linux Mint Debian Installer.

Click Forward.
Some final installation steps will now complete.
A pop-up, titled Installation finished, will appear. Click Yes.

Your computer will reboot and start LMDE.

References

Plex Media Server on Open Media Vault

martin@wimpress.com (Martin Wimpress) — Tue, 27 Aug 2013 21:11:43 +0000

I’ve recently started using Plex Media Server to handle most media streaming duties around the house. I run in on Open Media Vault 0.5.x. At the time of writing Open Media Vault is based on Debian (Squeeze) 6.0.

Plex Media Server

Anyway, it turns out that installing Plex Media Server on Open Media Vault is super simple thanks to the hard work of Christian Svedin who has packaged everything and made it available via an apt repository.

sudo apt-get install curl
echo "deb http://shell.ninthgate.se/packages/debian squeeze main" | sudo tee -a /etc/apt/sources.list.d/plexmediaserver.list
curl http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install plexmediaserver

The instructions above also work for Debian Wheezy, just change squeeze to wheezy in /etc/apt/sources.list.d/plexmediaserver.list.

When the install is complete point your browser at Plex/Web, for example:

http://plex.example.org:32400/web

Replace plex.example.org with your Plex Media Server hostname/ip address.

If you have PlexPass then head over to the Preview Releases for Debian and download and install the latest .deb.

Plex Clients

I use the Plex Client for Android on phone and tablet, a Roku 2 XS in the lounge and Roku 2 LT in the bedroom.

I’ve successfully tested Plex Home Theatre on Ubuntu 12.04 and Raspbmc with the PleXBMC plug-in on Raspberry Pi.

References

Setting up MiniDLNA on Open Media Vault

martin@wimpress.com (Martin Wimpress) — Mon, 26 Aug 2013 21:49:09 +0000

I have Open Media Vault running on a HP ProLiant MicroServer G7 N54L.

OpenMediaVault (OMV) is a network attached storage (NAS) solution based on Debian Linux. At the time of writing OMV 0.5.x is based on Debian 6.0 (Squeeze).

In recent months Plex has taken over just about all media streaming duties in our house, with the expectation of streaming music because Plex music playback and streaming is seriously lacking (no playlists!). So, MiniDLNA is still required for serving up music around the house.

Install MiniDLNA on OMV

There is a 3rd party plugin repository for Open Media Vault which includes packages to install MiniDLNA and a WebUI plugin for managing MiniDLNA via OMV. I upgraded to OMV 0.5.x this morning. and with the 0.5.x the Plugin API changed and as of the time of writing none of the 3rd party plugins had not been migrated to OMV 0.5.x. That said, MiniDLNA is super simple to configure so a WebUI is not a requirement for me.

MiniDLNA is not currently packaged for Debian Squeeze in the official repositories but Steve Kemp has packaged a fairly up-to-date version of MiniDLNA for Squeeze. Brilliant! Do the following as root to install MiniDLNA.

wget http://packages.steve.org.uk/minidlna/apt-key.pub
apt-key add apt-key.pub
echo deb http://packages.steve.org.uk/minidlna/squeeze/ ./" > /etc/apt/sources.list.d/minidlna.list
echo "deb-src http://packages.steve.org.uk/minidlna/squeeze/ ./" >> /etc/apt/sources.list.d/minidlna.list

Once you’ve done that run the following.

apt-get update
apt-get install minidlna

The MiniDLNA defaults in Steve’s package assume you have your music in /srv/music. So you’ll probably need to modify /etc/minidlna/minidlna.conf accordingly. From this point man minidlna and man minidlna.conf will guide you.

References

SketchUp Make on Arch Linux

martin@wimpress.com (Martin Wimpress) — Thu, 15 Aug 2013 15:42:47 +0000

I use SketchUp at work to manipulate models for use in Google Earth. Here is how I got SketchUp Make 2013 installed and working on Arch Linux under Wine 1.7.

Wine for Arch Linux

Install Wine on Arch Linux as follows.

sudo pacman -S --needed icoutils libwbclient libxslt lib32-mpg123 p11-kit lib32-p11-kit samba wine winetricks wine-mono wine_gecko
sudo packer -S --noedit --noconfirm ttf-ms-fonts

For 64-bit also install the following.

sudo packer -S --noedit --noconfirm lib32-libwbclient lib32-libxslt

Installing SketchUp Make

Once Wine is installed download SketchUp Make 2013.

http://www.sketchup.com/download

Create a clean wine prefix.

export WINEPREFIX="${HOME}/.sketchup"
export WINEARCH="win32"
wineboot

Install corefonts using winetricks

winetricks corefonts

Start the SketchUp Make setup.

wine SketchUpWEN.exe

Follow the installation wizard, I just went with the defaults.

That’s it. SketchUp is installed and should be associated with the appropriate file types.

Video Corruption

My workstation has a Radeon 5000 series graphics card and I use the Open Source radeon driver. I don’t know if this problem is specific to my hardware/drivers but SketchUp will eventually (sometimes immediately) encounter video corruption. Once that happens I can’t see or manipluate the models.

The solution that works for me is:

env WINEPREFIX="${HOME}/.sketchup" LIBGL_ALWAYS_SOFTWARE=1 vblank_mode=0 wine "C:\Program Files\SketchUp\SketchUp 2013\SketchUp.exe"

If this also works for you then SketchUp.desktop can be modified to persist these settings.

nano ~/.local/share/applications/wine/Programs/SketchUp\ 2013/SketchUp.desktop

Replace the contents with what follows but change USER with your username.

[Desktop Entry]
Name=SketchUp
Exec=env WINEPREFIX="/home/USER/.sketchup" LIBGL_ALWAYS_SOFTWARE=1 vblank_mode=0 wine C:\\\\windows\\\\command\\\\start.exe /Unix /home/USER/.sketchup/dosdevices/c:/users/USER/Start\\ Menu/Programs/SketchUp\\ 2013/SketchUp.lnk
Type=Application
StartupNotify=true
Icon=1871_SketchUpIcon.0

Uninstalling SketchUp Make

Should you ever need to, you can uninstall SketchUp Make as follows.

rm -rfv ${HOME}/.sketchup/
rm -rfv ~/.local/share/applications/wine/Programs/SketchUp\ 2013/

References

http://appdb.winehq.org/objectManager.php?sClass=version&iId=28620

Setting up Open Media Vault on the HP MicroServer N54L

martin@wimpress.com (Martin Wimpress) — Wed, 07 Aug 2013 21:21:09 +0000

I’ve installed Open Media Vault (OMV) on my new HP ProLiant MicroServer G7 N54L to replace my aging, and lackluster, ReadyNAS NV.

OpenMediaVault (OMV) is a network attached storage (NAS) solution based on Debian Linux. At the time of writing OMV 0.5.x is based on Debian 6.0 (Squeeze).

This blog post is not going to cover the extremely simple OMV installation procedure, it assumes OMV 0.5.x is already installed. This post explains how to upgrade the kernel, install some addtional plugins and some hackery to update Transmission.

This blog post is basically the essential notes I need to recreate my server setup.

N54L Custom BIOS

I’ve installed one of the custom BIOS mods for the N54L.

HP N36L/N40L/N54L Microserver Updated AHCI BIOS Support

I selected the BIOS mod above because the guy who created was an HP engineer and this BIOS mod only enables additional features that the N54L can actually support. Using this BIOS mod I’ve been able to:

Enable AHCI for the Optical Disk Drive (ODD) port.
Enable AHCI and port multiplier for the the e-SATA port.
Make all drives hot-pluggable.

The 250GB hard drive that came with N54L is now relocated in the optical drive day and being used as the OS drive, leaving all 4 bays for data.

As some point in the future I may want to hook up a 4 bay e-SATA enclosure and this BIOS mod makes that possible.

Open Media Vault

Once Open Media Vault is installed, I do the following.

Enable SSH

OMV actually has a really good WebUI that can be used to accomplish most update/upgrade tasks but I can’t help myself. I must have shell access. From the OMV WebUI:

Services -> SSH
- Put a tick in Enable and click the OK button.

Shell Tools

Things crave when at the a shell.

apt-get install less lsb-release rsync screen tree

OMV Plugins

OMV has a number of built-in plugins and a third party repository of plugins.

Built-in Plugins

Update the built-in plugins.

System -> Plugins
- Click the Check icon.

Logical Volume Manager

I use LVM. There, I said it. Enable the LVM2 plugin as follows from the OMV WebUI.

System -> Plugins
- Highlight the openmediavault-lvm2 plugin.
- Click the Install icon and then Yes.
- When the install is Done ..., click Close.

3rd Party OMV-Plugins

Follow the instructions on the following page to enable the OMV-Extras plugin repository.

http://omv-extras.org/simple/index.php?id=how-to-install-omv-extras-plugin

Backports 3.2 Kernel

I updated the Kernel to 3.2 because it better supports the N54L hardware, in particular the embedded graphics controller. The Linux 3.2 kernel can be installed via OMV-Extras.

System -> OMV-Extras.org -> Install Backports 3.2 kernel

Plex Media Server

Plex Media Server is available as a plugin once the OMV-Extras plugin repository is enabled. Plex is managed via the OMV WebUI.

Services -> Plex

Transmission

Transmission is available as a plugin once the OMV-Extras plugin repository is enabled. Transmission is managed via the OMV WebUI.

Services -> BitTorrent -> Server.

It’s all very straight forward. I use the following block list.

http://list.iblocklist.com/?list=bt_level1&fileformat=p2p&archiveformat=gz

That pretty much covers the stuff I won’t remember in the future. I’m considering adding LXC and Dropbox in the coming that will require some manual steps.

References

Basic Debian Setup for Squeeze and Wheezy

martin@wimpress.com (Martin Wimpress) — Tue, 06 Aug 2013 20:07:09 +0000

Recently I’ve been deploying Debian 6.0 (Squeeze) and 7.0 (Wheezy) servers for some personal projects. These servers are provisioned in different ways:

Open Media Vault using a Squeeze pre-seed
VPS powered by LXC deployed via debootstrap
VPS powered by KVM using the hosting providers Wheezy pre-seed

Consequently the basic install differs on each instance and requires a little bit of post install tweaking to get them all consistent. This blog post is a quick reference for the post install steps I complete on Debian servers.

Timezone & Locale

Select your timezone.

dpkg-reconfigure tzdata

Select your locale(s).

dpkg-reconfigure locales

Make sure the locales are correctly generated. Replace en_GB.UTF-8 with your default locale.

update-locale LANG=en_GB.UTF-8 LANGUAGE=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_TIME=en_GB.UTF-8 LC_CTYPE=en_GB.UTF-8
locale -a
locale-gen

Hostname

echo box.example.org > /etc/hostname
/bin/hostname -F /etc/hostname

Update /etc/hosts accordingly.

Time

Keeping time is essential.

apt-get install ntp ntpdate

Force a clock sync.

service ntp stop
ntpdate -s pool.ntp.org
service ntp start

If your VPS is a Xen DomU then checkout the following.

http://jinntech.blogspot.co.uk/2009/03/xen-and-keeping-time.html

Essentials

These are the essential tools I require.

apt-get install build-essential curl git htop less lsb-release nano \
rsync screen sudo tree whois

Users

The following will create a user with sudo capabilities.

useradd user_a --create-home --shell /bin/bash --user-group \
--groups adm,dialout,cdrom,plugdev,sudo

This will create a regular user.

useradd user_b --create-home --shell /bin/bash --user-group --groups adm,dialout,cdrom,plugdev

Assign a password.

echo user_a:mypassword | chpasswd

An existing user can be made a sudoer by simply adding them to the sudo group.

adduser user_b sudo

Firewall

I use firewall my VPS server with ufw. This is my initial configuration that allow access via SSH only.

sudo apt-get install ufw

Configuring ufw is simple but make sure you have console access to the host you are configuring just in case you lock yourself out.

NOTE! When enabling ufw the chains are flushed and connections may be dropped. You can add rules to the firewall before enabling it however, so if you are testing ufw on a remote machine it is recommended you perform…

ufw allow ssh/tcp

…before running sudo ufw enable. Once the firewall is enabled, adding and removing rules will not flush the firewall, although modifying an existing rule will.

Set the default behaviour to deny all incoming connections.

sudo ufw default deny

Open up TCP port 22 but with rate limiting enabled which will deny connections from an IP address that has attempted to initiate 6 or more connections in the last 30 seconds. Ideal for protecting sshd but you should conisder other SSH brute force defense techniques as well.

sudo ufw limit ssh

To enable the firewall you also have to do the following.

sudo ufw enable

On low-end servers it might be beneficial to disable logging.

sudo ufw logging off

You can see the status of the firewall using sudo ufw status.

Intrusion prevention

I use either denyhosts

sudo apt-get install denyhosts

Purge entries older than 5 days, denied hosts will only be purged twice and disable email alerts.

sudo sed -i 's/#PURGE_DENY = 5d/PURGE_DENY = 5d/' /etc/denyhosts.conf
sudo sed -i 's/#PURGE_THRESHOLD = 2/PURGE_THRESHOLD = 2/' /etc/denyhosts.conf
sudo sed -i 's/root@localhost//' /etc/denyhosts.conf

Restart denyhosts.

sudo service denyhosts restart

Also see SSH brute force defence.

Boot options

These servers are headless and often remote, therefore I enable fsck auto repair.

Squeeze

sed -i 's/FSCKFIX=no/FSCKFIX=yes/' /etc/default/rcS

Wheezy

sed -i 's/#FSCKFIX=no/FSCKFIX=yes/' /etc/default/rcS

Repositories

lsb-release was installed earlier. This is what I put in /etc/apt/sources.list.

cat >/etc/apt/sources.list<<EOF
deb http://ftp.uk.debian.org/debian/ $(lsb_release -cs) main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ $(lsb_release -cs) main contrib non-free

deb http://security.debian.org/ $(lsb_release -cs)/updates main contrib non-free
deb-src http://security.debian.org/ $(lsb_release -cs)/updates main contrib non-free

# $(lsb_release -cs)-updates, previously known as 'volatile'
deb http://ftp.uk.debian.org/debian/ $(lsb_release -cs)-updates main contrib non-free
deb-src http://ftp.uk.debian.org/debian/ $(lsb_release -cs)-updates main contrib non-free
EOF

I replace ftp.uk with ftp.us for servers located in the United States.

sed -i 's/ftp\.uk/ftp\.us/g' /etc/apt/sources.list

Backports

I add the Backports repository in order to access some updated packages.

Squeeze

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.uk.debian.org/debian-backports $(lsb_release -cs)-backports main contrib non-free
deb-src http://ftp.uk.debian.org/debian-backports $(lsb_release -cs)-backports main contrib non-free
EOF

Wheezy

cat >/etc/apt/sources.list.d/backports.list <<EOF
deb http://ftp.uk.debian.org/debian $(lsb_release -cs)-backports main contrib non-free
deb-src http://ftp.uk.debian.org/debian $(lsb_release -cs)-backports main contrib non-free
EOF

Update.

sudo apt-get update

All backports are deactivated by default (i.e. the packages are pinned to 100 by using ButAutomaticUpgrades: yes in the Release files. If you want to install something from backports run:

apt-get -t wheezy-backports install "package"

MTA

sSMTP is a simple MTA to deliver mail from a computer to a mail hub. sSMTP is simple and lightweight.

Remove exim4

Some VPS Debian templates from VPS providers have exim4 installed and running by default. Remove it.

sudo service exim4 stop
sudo apt-get purge exim4 exim4-base exim4-config

Install sSMTP

apt-get install ssmtp bsd-mailx

sSMTP Gmail Configuration

I use Gmail as my smart host, here is an example configuration for /etc/ssmtp/ssmtp.conf.

#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=root@example.org

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
mailhub=smtp.gmail.com:587

# Where will the mail seem to come from?
rewriteDomain=

# The full hostname
hostname=box.example.org

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address
FromLineOverride=YES

# Gmail requires TLS
UseTLS=YES
UseSTARTTLS=YES

# Username and password for Gmail servers
AuthUser=yourgmailname@gmail.com
AuthPass=youpassword
AuthMethod=LOGIN

Then add each account that you want to be able to send mail from by editing /etc/ssmtp/revaliases:

root:username@gmail.com:smtp.gmail.com:587
user_a:username@gmail.com:smtp.gmail.com:587
user_b:username@gmail.com:smtp.gmail.com:587

https://wiki.debian.org/sSMTP

Log and package monitoring

My personal VPS server are dotted about the place but I like to keep an eye on them and I find apticron and logwatch are very useful for that.

apticron

apticron is a simple tool to mail about pending package updates.

sudo apt-get install apticron

logwatch

Logwatch is a modular log analyser that runs every night and mails you the results.

sudo apt-get install logwatch

Lighter

Some of my servers have fairly low resources, these are some simple changes that can save a bit of RAM or disk space.

aptitude

I don’t use it.

sudo apt-get purge aptitude

D-Bus

D-Bus is a message bus, used for sending messages between applications. Some VPS provider Debian templates have D-Bus and avahi installed. I don’t require these on Internet facing servers so I remove them. If an application pull in D-Bus as a requirement that is fine, but for this initial server state I remove it.

sudo apt-get purge dbus

at

at provides delayed job execution and batch processing. I don’t use it.

sudo service atd stop
sudo apt-get purge at

ngetty

Ngetty is a single-process getty replacement, so instead of running 6 getty processes consuming up to 3MB of RAM each, you can use a single ngetty process using less than 1MB of RAM total.

sudo apt-get install ngetty

Edit /etc/inittab, comment out getty and add ngetty like so.

#1:2345:respawn:/sbin/getty 38400 tty1
#2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6
ng:2345:respawn:/sbin/ngetty 1 2 3 4 5 6

Restart inittab

telinit q

http://haydenjames.io/replacing-getty-ngetty-debian/

That about covers the general post installation step I complete on my Debian servers.

Clean up.

Remove any packages that are no longer required and clean up the package cache.

sudo apt-get autoremove
sudo apt-get autoclean
sudo apt-get clean

NFS and CIFS mounts with systemd

martin@wimpress.com (Martin Wimpress) — Fri, 26 Jul 2013 12:00:00 +0000

I have an NFS server at home and at work we have Windows (not Samba) servers. When I first switched to systemd I noticed that boot and shutdown were seriously delayed while NFS and CIFS were mounted/unmounted. systemd was designed to eliminate those kinds of delays, so I did some research to find out how to correctly mount NFS and CIFS using systemd.

systemd friendly fstab

Below are some example /etc/fstab entries for NFS and CIFS mounts that are systemd friendly, the pertinent mount options are:

noauto,x-systemd.automount,_netdev

I found that noauto,x-systemd.automount improved the boot performance and _netdev improved the shutdown performance.

NFS

This is typically what I have use for mounting my home NAS.

nfs-server:/SomeData /media/SomeData nfs defaults,noauto,x-systemd.automount,_netdev,noatime 0 0

CIFS

This is what I use at work to work correctly with Windows Server.

//cifs-server/MoreData /media/MoreData cifs defaults,noauto,x-systemd.automount,_netdev,rw,noperm,credentials=/home/username/.smb-credentials 0 0

The contents of the credentials file looks something like this.

username=yourusername
password=yourpassword
domain=COMPANYDOMAIN

References

https://wiki.archlinux.org/index.php/Systemd#Filesystem_mounts

Python Distributions

martin@wimpress.com (Martin Wimpress) — Mon, 24 Jun 2013 13:11:43 +0000

At work I maintain the Jenkins test and build servers. I’m just about to update our Windows build servers and thought I’d better check the available “Python Distributions” to see if our current choice (the brilliant Python(x,y) is still the most suitable for our needs.

Our Flight Data Analyzer makes extensive use of Numpy, Scipy, h5py and other analysis tools. So, pre-built Python distributions on Windows save me a lot of ~~pain~~ time. On Linux we roll our own of course.

What follows is a list of Python Distributions that include Python and the essential modules we require.

Anaconda

Completely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing.

https://store.continuum.io/cshop/anaconda/

Heard about this for the first time a couple of days ago. It looks very promising with 32-bit and 64-bit flavours and MKL optimised modules are available from the reasonably priced Anaconda Accelerate. While I roll my own Python Distribution for our Linux build servers, I am rather taken with the idea of using Anaconda on Linux and Windows to provide a consistent platform everywhere. I’m looking forward to testing Anaconda this week.

Continuum appear to be taking on Enthought at their own game, and good luck to them as they have some really interesting projects on the go.

http://www.continuum.io/developer-resources

Enthought Canopy

Enthought Canopy is a comprehensive Python analysis environment with easy installation & updates of the proven Enthought Python distribution - all part of a robust platform you can explore, develop and visualize on.

https://www.enthought.com/products/canopy/

We used to use Entought EPD and Canopy builds on EPD. However, we decided to switch from EPD and consolidate analyst workstation and build server deployments around Python(x,y).

There were several factors to this decision, but the main issue was that updates to the EPD package repositories were slow for some essential modules we use. Canopy seems to have inherited package latency from EPD as Numpy is still at 1.6.1 while we now require Numpy 1.7.

Paid versions of Canopy have MKL optimizations and 64-bit platform support.

https://www.enthought.com/products/canopy/compare-subscriptions/

Python(x,y)

Scientific-oriented Python Distribution based on Qt and Spyder.

https://code.google.com/p/pythonxy/

This is what we currently use for Windows build servers and analyst workstations. The only reason I’m conisdering switching is that is it 32-bit only. Other than that, I love it and highly recommend it. Python(x,y) doesn’t offer MKL optimisations.

WinPython

WinPython is a portable scientific Python 2/3 32/64bit distribution for Windows

http://code.google.com/p/winpython/

From the same stable as Python(x,y) but has 32-bit and 64-bit flavours, yummy. WinPython includes everything I need so will definately get fully tested this week.

Portable Python

Portable Python is a Python programming language pre-configured to run directly from any USB storage device, enabling you to have, at any time, a portable programming environment.

http://portablepython.com/

Not looked at this in any real detail. Appears to be 32-bit only but does include a number of essential packages.

Unofficial Windows Binaries for Python Extension Packages

Provides 32- and 64-bit Windows binaries of many scientific open-source extension packages for the official CPython distribution of the Python programming language.

http://www.lfd.uci.edu/~gohlke/pythonlibs/

OK, so this is not a Python distribution but it is compelling. 32-bit and 64-bit platforms are catered for and MKL optimizations are available at no cost. Each package needs to be installed individually, which can be seen as both good and bad. Good because you only install what you actually require and bad because the initial installation is protracted. That said, it is on my evaluation list for this week.

Anymore?

Those are the Python Distributions I’m aware of. Are there any others I should consider?

Spring cleaning Arch Linux

martin@wimpress.com (Martin Wimpress) — Tue, 07 May 2013 18:09:21 +0000

About a year ago I migrated all my workstations, laptops and netbooks to Arch Linux. Since then, I’ve setup Arch Linux on a Raspberry Pi and this server was also recently migrated to Arch Linux.

I’ve had no major issues issues during the last year and have upgraded through five major Linux kernels, transitioned to systemd and upgraded from GNOME 3.2 to 3.8.

Although I have been disciplined about merging .pacnew files frequently, during the upgrades and experimentation’s I have packages installed that I no longer require and obsolete files kicking about.

After the upgrade to GNOME 3.8 I decided to clean up a little. I rarely dip into the AUR, but when I do I always use packer to clearly separate what is official from what is not.

Finding what is installed

The following commands are useful for identifying installed packages based on where they were installed from. The package lists generated from the commands below can be quite big but often highlight packages that I know I’m no longer using nor require.

Listing installed packages

List packages installed from the official repositories.

pacman -Qq | grep -Fv -f <(pacman -Qqm)

List packages installed from the AUR.

pacman -Qqm

Listing installed packages by size

Use pacsysclean to list installed packages sorted by size, it helps identify large packages that are no longer required which can the be manually uninstalled.

Listing orphaned packages

List orphaned packages install from the official repositories.

pacman -Qqtd | grep -Fv -f <(pacman -Qqtdm)

List ophaned packages from the AUR.

pacman -Qqmtd

Getting package information

Get package information for a package in the official repositories.

pacman -Si <package>

Get package information for a package in the AUR.

packer -Si <package>

Removing orphaned packages

Removing orphaned packages manually can be very time consuming, but is by far the safer option. However, I decided to take a brave pill a uninstall all orphaned packages automatically.

Remove all orphaned packages installed from the official repositories.

sudo pacman -Rs `pacman -Qqtd | grep -Fv -f <(pacman -Qqtdm)`

Remove all ophanced packages install from the AUR.

sudo pacman -Rs $(pacman -Qqtdm)

Re-installing what you do need

When you do something scary like removing all the obsolete packages automatically, then you should really make sure you do have everything install that you require.

Re-install 64-bit base

sudo pacman -S --needed `pacman -Sqg base multilib-devel | grep -v gcc-libs | tr '\n' ' '`

Re-install 32-bit base

sudo pacman -S --needed `pacman -Sqg base base-devel | tr '\n' ' '`

Reinstall the groups required for a GNOME 3 desktop.

sudo pacman -S --needed `pacman -Sqg gnome gnome-extra telepathy | tr '\n' ' '`

Install all missing dependencies for packages in the official repositories.

sudo pacman -S --needed `pacman -Si $@ 2>/dev/null | awk -F ": " -v filter="^Depends" \ '$0 ~ filter {gsub(/[>=<][^ ]*/,"",$2) ; gsub(/ +/,"\n",$2) ; print $2}' | grep -v smtp- | sort -u`

Install all missing dependencies for packages in the AUR. This will re-install even if the package is already installed. I can’t be arsed to filter it out for a one liner.

sudo packer -S --noedit --noconfirm `packer -Si $(pacman -Qqm) 2>/dev/null | awk -F ": " -v filter="^Depends" \ '$0 ~ filter {gsub(/[>=<][^ ]*/,"",$2) ; gsub(/ +/,"\n",$2) ; print $2}' | grep -v java- | sort -u`

Find files not associated with a package

When packages are removed they may leave some files behind. The following will find all files not associated with a package. These files can not be automatically deleted, each entry requires assessment.

pacman -Qlq | sort -u > /tmp/db
sudo find /bin /etc /sbin /usr ! -name lost+found \( -type d -printf '%p/\n' -o -print \) | sort > /tmp/fs
comm -23 /tmp/fs /tmp/db

As with all spring cleaning chores, I got bored by this stage as my workstation was looking pretty tidy. Much of what is presented in this blog post is a rehash of what others have already contributed to the Arch Linux Wiki. I’ve just organised what “Works For Me ™” so I know what to do next year.

References

https://wiki.archlinux.org/index.php/Pacman_Tips

get-iplayer on Debian 6.0

martin@wimpress.com (Martin Wimpress) — Mon, 25 Mar 2013 18:09:21 +0000

I use get-iplayer to download TV programs so I can watch them on the devices that suit me, when it suits me. What follows is how I install get-iplayer on a headless Debian 6.0 server I have a home. The server in question is really low powered so building from source was not an option.

In order to install the latest version of get-iplayer (currently 2.82) on Debian Squeeze a couple of additional package repositories need enabling.

Enable the Debain Backports repository by adding the following line to /etc/apt/sources.list.d/backports.list.

deb http://backports.debian.org/debian-backports squeeze-backports main

Enable the Debain Multimedia repository by adding the following lines to /etc/apt/sources.list.d/multimedia.list.

deb http://www.deb-multimedia.org squeeze main non-free
deb http://www.deb-multimedia.org squeeze-backports main

Update the repositories.

sudo apt-get update

Install the deb-multimedia-keyring package.

sudo apt-get --allow-unauthenticated install deb-multimedia-keyring

Install get-iplayer (currently v2.78) from the official Debian repositories, this will also install the dependencies.

sudo apt-get install get-iplayer

Install the get-iplayer suggested packages.

sudo apt-get install ffmpeg rtmpdump libdata-dump-perl libid3-tools libcrypt-ssleay-perl libio-socket-ssl-perl

I have seen it suggested that mplayer should also be installed. I’ve not determined if that is an absolute requirement. But this is how to install it on a headless Debian computer.

sudo apt-get --no-install-recommends install mplayer

Finally, upgrade get-iplayer to v2.82.

sudo apt-get install libmp3-tag-perl libxml-simple-perl
wget http://ftp.uk.debian.org/debian/pool/main/g/get-iplayer/get-iplayer_2.82-2_all.deb
sudo dpkg -i get-iplayer_2.82-2_all.deb

At this point get-iplayer should be good to go and the get-iplayer website and man get-iplayer will assist you.

References

Ployer Momo8 IPS Custom Firmware

martin@wimpress.com (Martin Wimpress) — Fri, 22 Mar 2013 13:35:34 +0000

I bought a budget Android tablet a little while back that turned out to be really rather good. However, there were issues with the initial firmware.

Connecting to, or maintaining connection with, some wireless networks was unreliable.
When the internal NAND was under moderate load the tablet would become unresponsive.

Ployer released a firmware update in November 2012 and again in April 2013 which addressed these issues. Here’s the translated change log.

Increase the volume buttons on the vertical screen.

System, audio and video decoding, browser, Flash player, 3G module, boot animation module BUG repair.

Update NandFlash, Mali, Wifi module drivers, while addressing some CTS tests BUG and the stability of the system as a whole has been further enhanced.

Optimize Flash stability, improve the efficiency of the implementation of the DDR.

Default input method Sogou input method.

Update Google Pinyin input method.

However, the official Ployer firmware comes pre-loaded with a selection of Chinese applications and defaults to a Chinese language.

Objectives

I decided to have a go a making my own custom firmware for the Ployer Momo8 IPS with the following goals.

Pre-rooted.
Removal of pre-installed Chinese apps.
Removal of bloatware.
Extend /data partition.
Make it a “Google Experience” device.
ClockWorkMod Recovery (available to donors)

I think I’ve been fairly successful. My firmware includes Android 4.1.2 features and even some Android 4.2 features. Until someone successfully ports cyanogenmod to the Ployer Momo8 IPS or Ployer release a 4.2.x update then my firmware is the most complete “Google Experience” you’ll find for the device.

Changes

Below are the changes I made to the official Ployer Momo8 IPS firmware.

13.164

This release is mainly a fix for Google+. The builds are running now and will be available for download later today.

Integrated SuperSU 1.32
- http://forum.xda-developers.com/showthread.php?t=1538053
Tweaked WiFi 2.4Ghz Transmit Power. maxp2ga0 is now 76 in:
- This appears to be the default for other devices with the same chipset, feedback welcomed.
Reverted PlusOne.apk to the version in 13.137
- This should fix the force closes.

13.149

The purpose of this release is to refresh some system apps, address some compatibility issues and to offer versions of my custom firmware based on both 20130325 and 20121120.

Based on the Official Ployer MOMO8(IPS)-4.1.1-Firmware-20130325 and the Official Ployer MOMO8(IPS)-4.1.1-Firmware-20121120.
- Feedback suggests that WiFi works better for some when using firmware based on the 20121120 version.
- Only donors will have access to the 20121120 versions.
Integrated SuperSU 1.30
- http://forum.xda-developers.com/showthread.php?t=1538053
Removed the framework.jar patch as it introduces incompatibilities of its own.
Removed the PicoTTS voice data files from the /system partition.
- PicoTTS voices can be selectively installed via Settings -> Language and Input.
- This is a space saving measure and essential for the 20121120 version.
Updated the following, as of May 25th 2013:
- Magazines.apk (Google Play Magazines)
- Music2.apk (Google Play Music)
- PlusOne.apk (Google+)
- Talk.apk (Hangouts)

13.137

This release is focused on stability, adding the new Google Play features announced at Google I/O 2013 and also includes everything from the previous releases.

Integrated Android 4.2 sounds, fonts and boot animation
- Boot animation is now the correct size, higher quality and doesn’t flicker at the end of a boot.
- All Android 4.2 sounds and fonts are now included.
  - Fixed default notification sound.
  - Lock screen font looks much nicer.
- http://forum.xda-developers.com/showthread.php?t=1991734
Integrated Android 4.2.2 keyboard
- http://forum.xda-developers.com/showthread.php?p=38124560
Integrated Nova Launcher 2.1
- http://novalauncher.com
Integrated Feedly 15.0.1
- http://www.feedly.com
Integrated Power Toggles 4.6.6
- http://www.powertoggles.com
Tweaked WiFi 2.4Ghz Transmit Power. maxp2ga0 is now 100 in:
- /etc/firmware/nvram_RK901.txt was previously 74.
- /etc/firmware/nvram_RK903.cal was previously 76.
- /etc/firmware/nvram_RK903.txt was previosuly 72.
- /etc/firmware/nvram_RK903_26M.cal was previously 60.
Tweaked WiFi 5.0Ghz Transmit Power. maxp5ga0, maxp5gla0 and maxp5gha0 are now 100 in:
- /etc/firmware/nvram_RK903.cal were all previously 80.
- /etc/firmware/nvram_RK903_26M.cal were all previously 80.
Added camera icon to the application drawer.
Fixed boot up time
- Boot up time is significantly faster, with the exception of the first boot after a firmware flash.
Fixed apps disappearing from the Settings and the Play Store.
Replaced Google Talk with Hangouts
Removed Google Drive and Google Earth
- When pre-installed they would Force Close.
Updated the following, as of May 16th 2013:
- Books.apk (Google Play Books)
- GestureSearch.apk
- Currents.apk
- GoogleEars.apk (Sound Search)
- GmsCore.apk (Google Play Services)
- Magazines.apk (Google Play Magazines)
- Maps.apk
- Music2.apk (Google Play Music)
- Phonesky.apk (Play Store)
- Talk.apk (Hangouts)
- TalkBack.apk (Accessibility)
- Videos.apk (Google Play Movies)

13.129

This version includes everything from the previous release plus the changes below.

Based on the Official Ployer MOMO8(IPS)-4.1.1-Firmware-20130325
- http://download.ployer.cn/downdetail.asp?id=763
Boot loader upgraded to 1.22
- Switching to Upgrade mode in Rockchip Batch Tool is much quicker.
Extended /system partition from 375M to 428M
Removed more Chinese applications and bloatware
- Removed UCBrowser shared objects from /system/lib
- Removed QQMiniHD shared objects from /system/lib
- Removed more Adobe Reader shared objects from /system/lib and fonts from /system/fonts/adobefonts
Integrated Clockworkmod Recovery to 6.0.31 (donors only)
- http://androtab.info/clockworkmod/rockchip/changelog/
Integrated Adobe Flash 11.1.115.54
- http://helpx.adobe.com/flash-player/kb/archived-flash-player-versions.html
Integrated File Wrangler 1.5
- https://play.google.com/store/apps/details?id=com.amon.filewrangler
Integrated AdAway 2.3 (Google now blocks all advert blockers from the Play Store)
- http://code.google.com/p/ad-away/
Replaced Launcher2 4.1.5 with Nova Launcher 2.0.2
- http://novalauncher.com
Replaced the default wallpaper with the default wallpaper from the Nexus 10.
Removed User Management - it was not reliable.
Removed Chrome.apk but updated /system/lib/libchromeview.so from Chrome 26.0.1410.58
- This is a space saving measure but ensures the ROM is Chrome compatible.
Updated the following, as of May 10th 2013:
- Authenticator.apk
- CalendarGoogle.apk
- Currents.apk
- Drive.apk
- Earth.apk
- Gmail.apk
- GooglePlayBooks.apk
- GooglePlayMagazines.apk
- GooglePlayMovies.apk
- GooglePlayMusic.apk
- GooglePlayServices.apk
- Keep.apk
- Maps.apk
- Phonesky.apk (Play Store)
- PlusOne.apk (Google+)
- Street.apk
- YouTube.apk

13.080

Based on the Official Ployer MOMO8(IPS)-4.1.1-Firmware-20121120
- http://download.ployer.cn/downdetail.asp?id=763
Extended /data partition to 2GB.
- http://www.freaktab.com/showthread.php?287-RockChip-ROM-Building-Tips-and-Tricks-by-Finless&p=4054&viewfull=1#post4054
Removed Chinese applications.
- UCBrowser_V2.1.1.219_Android3_pf147_(Build12110718).apk
- dopoolplayer
- iReader_android_v2000_108044_guanwang.apk including shared objects from /system/lib.
- market_hd.apk
- qiyiyingshi_V3.0_mumayi_e3d3e.apk
- qq_mini_hd_1.9.1.apk
- sougoushurufa.apk including shared objects from /system/lib.
- zhonghuawannianli_ECalendar_V3.2.3_mumayi_3ee39.apk
Cleaned up build.prop
- Changed Chinese strings to ASCII
- Configured English (UK) as default language/locale.
- Configured Europe/London as default time zone.
- Default display brightness set at 50%
- Deleted settings for Chinese applications that have been removed.
Removed Rockchip Utilities.
- ApkInstaller.apk
- RkExplorer.apk
- RKUpdateService.apk including shared objects from /system/lib.
- RkVideoPlayer.apk
Removed bloatware.
- Adobe Reader including shared objects from /system/lib.
- DocsToGo.
Integrated SuperSU 1.25
- http://forum.xda-developers.com/showthread.php?t=1538053
Integrated Clockworkmod Recovery 6.0.28
- http://androtab.info/clockworkmod/rockchip/
- http://forum.xda-developers.com/showthread.php?t=2102679
Integrated Google Apps (JZO54K) 4.1.2
- Adds the first boot Setup Wizard.
- Adds Picasa photo syncing.
- Face Unlock 4.1.2
- Google Talk 4.1.2
- http://rootzwiki.com/topic/31532-gapps-412-485486-jzo54k-gapps-package-1012/
Integrated Launcher2 4.2, includes 50% transparency on app drawer.
- http://forum.xda-developers.com/showthread.php?t=1995812
Integrated some Jelly Bean 4.2 features:
- Jelly Bean 4.2 clock.
- Jelly Bean 4.2 keyboard with gesture.
- Movie Studio video editor.
- http://forum.xda-developers.com/showthread.php?t=2010535
Integrated Cyanogenmod 10.0 APKs
- Calculator.apk : Adds different calculator panels.
- Galaxy4.apk : Smaller APK
- HoloSpiralWallpaper.apk : Smaller APK
- LiveWallpapers.apk : Smaller APK
- LiveWallpapersPicker.apk : Smaller APK
- MagicSmokeWallpapers.apk : Smaller APK
- NoiseField.apk : Smaller APK
- PhaseBeam.apk : Smaller APK
- SoundRecorder.apk : Smaller APK
- VisualizationWallpapers.apk : Smaller APK
Integrated Adobe Flash 11.1.115.47
- http://helpx.adobe.com/flash-player/kb/archived-flash-player-versions.html
Integrated Quick Boot 4.2
- https://play.google.com/store/apps/details?id=com.siriusapplications.quickboot
Integrated File Wrangler 1.4
- https://play.google.com/store/apps/details?id=com.amon.filewrangler
Integrated User Management
- http://forum.xda-developers.com/showthread.php?t=1824066
- https://play.google.com/store/apps/details?id=com.appaholics.um
Integrated busybox 1.20.2
Added additional permissions files for increased compatibility.
- android.hardware.camera.autofocus.xml
- android.hardware.camera.front.xml
- android.hardware.ethernet.xml
- android.hardware.sensor.accelerometer.xml
- android.hardware.sensor.light.xml
- android.hardware.touchscreen.multitouch.xml
Replaced ring tones, alerts and notifications with the audio from Nexus 4.
Replaced boot animation with the Nexus animation.
Replaced the default wallpaper with one from the Jelly Bean SDK.
Updated the following, as of March 18th 2012:
- CalendarGoogle.apk
- Gmail.apk
- GooglePlayMovies.apk
- GooglePlayMusic.apk
- GooglePlayServices.apk
- PlusOne.apk (Google+)
- Maps.apk
- Phonesky.apk (Google Play Store)
- Street.apk
- YouTube.apk
Removed Chrome.apk but updated /system/lib/libchromeview.so.
- This is a space saving measure but ensures the ROM is Chrome compatible.
Patched framework.jar so that Gameloft titles work.
- http://www.slatedroid.com/topic/50354-gameloft-game-fix-for-license-check-or-loop/
- http://dragondevs.com/index.php?/topic/1265-gameloft-game-fix-for-license-check-or-loop/
- http://forum.xda-developers.com/showthread.php?t=955847
All APKs are ZipAligned.

What didn’t change

I haven’t modified build.prop to make the Ployer Momo8 IPS masquerade as another brand or model of Android device. I will not be making this change, please don’t request it.
Although my firmware includes many features from Android 4.1.2 and some from Android 4.2 I haven’t bumped the Android version or Build number. That would be dishonest and misleading.

Known Issues

Adobe Flash is enabled the first time the Browser app (not Chrome) is executed.
Hangouts app does not work on the first boot following a firmware flash. It does works correctly on subsequent boots.
Although busybox is included in the ROM, no sym-links are created.
Some Gameloft titles may not work, Asphalt 7 for example. Do other Gameloft titles work properly? This problem is present in the official Ployer firmware too, there are framework patches to fix this problem but they seem to introduce other incompatibilities.

Donate

Please consider donating to this project. It is nice to have the effort I’ve put into this custom firmware recognized. I don’t ask for much, it is at your discretion, but just think how happy I’ll look when I am sipping the beer you bought me 😄

Downloads

You’ll need Rockchip Batch Tool which include the RockChip USB drivers and the firmware flashing utility. I’ve modified Rockchip Batch Tool to default to English language and removed old logs and transient data to reduce the size of the download. You’ll also need the firmware itself.

Ployer MOMO8 IPS Firmware

Download Rockchip Batch Tool and the STOCK RECOVERY version of the ROM. Unzip both archives once they are downloaded.

How to Flash

Warning Always (I can't stress this enough) flash the STOCK RECOVERY version of my custom ROM. Never flash the CWM RECOVERY version without first having flashed the STOCK RECOVERY version. If you are upgrading from another ROM (even one of mine) always flash the STOCK RECOVERY version first.

Flashing this ROM will will effectively factory reset your tablet, wipe your installed apps, app data and preferences. If you have rooted your tablet you might want to consider making a backup with Titanium Backup * root. If you have not rooted your tablet then you could use Helium - App Sync and Backup.

Install the Rockchip USB drivers included with Rockchip Batch Tool.
- This was simple of Windows XP but I couldn’t get Windows 7 to accept the Rockchip drivers. However, installing MoboRobo on Windows 7 provided the correct driver.
Turn the Ployer Momo8 IPS on.
On the tablet go to Settings -> Developer options and untick USB debugging.
Start RKBatchTool.exe.
In RKBatchTool choose firmware file, click the Switch button. The device icon should change to green to indicate a successful connection.

Note The very first time your `Switch` to upgrade mode, Windows may prompt you to install additional Rockchip USB drivers.

Only if the device icon changed to Green, click Upgrade.
The firmware will be flashed and the tablet will automatically reboot into Recovery.
You will see a green Android and then the paritions (/data, /cache and /mnt/sdcard) will be formatted.
- /mnt/sdcard is the internal memory, not the microSDHC card inserted in the card reader.
When the formatting is complete the tablet will reboot.

The first boot may take a little longer than usual. You will be presented with the Welcome wizard where you can configure language and locale, etc. You can optionally enter your Google Apps or Gmail account credentials and doing so will prompt for which Wifi network to associate with.

Rockchip Batch Tool Video

I’ve also made a video showing how to flash the firmware. Frankly, the hardest part is getting the Rockchip drivers installed.

How to Flash the CWM version

Warning Like I said, never flash the CWM RECOVERY version of my ROM without first having flashed the STOCK RECOVERY version. The CWM version is made available to donors.

Flash the STOCK RECOVERY version of the ROM as detailed above.
Follow the same procedure, but this time flash the CWM RECOVERY version.
When the CWM RECOVERY flash is complete the tablet will boot into CWM recovery. Do the following:
- wipe cache partition
- advanced -> wipe dalvik cache
- ++++++Go Back++++++
- reboot system now
The tablet will reboot, it will be a slower boot than usual but subsequent boots will be quicker.

ClockWorkMod Recovery

Booting to recovery can be achieved using the pre-install Quick Boot app.

The Ployer Momo8 IPS only has one hardware button (power) so CWM is controlled with gestures.

Swipe up/down: up/down
Swipe left: select
Swipe right: back

The Power button also acts as select, which I find to be the most reliable way to select an action.

FAQ

Does this custom firmware fix WiFi connectivity?

Short answer, possibly.

Of all the WiFi networks I have access to, only one causes the Ployer Momo8 IPS to encounter weak signal and intermittent connections. When using the Official Ployer firmware on that WiFi network, connecting more than 5 meters from the access point is unreliable and maintaining a connection is almost impossible.

Using my firmware I can connect and maintain a connection up to about 10 meters from the access point. My testing, and the feedback from others suggests the my firmware improves the WiFi signal by 8 to 10dBm.

From my testing the Momo8 IPS appears to have issues with some wireless access points, possibly related to the chipset in the access point or the Momo8 IPS, but I’ve not been able to pin it down. That said, I’ve got access to 2 Draytek Vigor routers and the Momo8 IPS does not work well with either of them. Every other wireless network I’ve connected to works well.

For example, at home I stream 720p movies via DLNA over WiFi and watch them on the Momo8 IPS using MX Player. It works perfectly and never buffers or lags. However, your mileage may vary.

Why should I always flash the STOCK RECOVERY version first?

Short answer, it’s the safest option.

There are 6 different ROMs (official or otherwise) that I am aware of for the Ployer Momo8 IPS. They all have slightly different partition layouts. If you flash one of my CWM RECOVERY ROMs over a ROM using a slightly different partition layout, it will almost certainly soft brick the tablet. The STOCK RECOVERY will always correctly format the partitions directly after a flash, thereby mitigating the risk of soft bricking the tablet.

Ooops, I’ve soft bricked my tablet. What do I do?

Read the comments here, there are several useful tips. You can also use the comments here to see if anyone can offer assistance. Alternatively, go the Ployer Momo8 forum on Slatedroid, read what others have done and ask for help.

http://www.slatedroid.com/forum/445-momo8/

How do the 20130325 and 20121120 versions differ.

The differences Ployer made between 20130325 and 20121120 is not well documented. I did some additional analysis of what changed between the November 2012 and April 2013 releases from Ployer, you can find my notes here:

New Official Firmware: MOMO8 (IPS) -4.1.1 firmware 20130325

In addition to whatever Ployer changed, this is how my 20121120 differs from 20130325.

The /system partition on the 20121120 version is 375MB, therefore the following system apps are not included:
- AdAway.apk
- Authenticator.apk
- Books.apk
- Currents.apk
- Feedly.apk
- GestureSearch.apk
- Keep.apk
- Magazines.apk
- Videos.apk
- YouTube.apk
- All of the above can be installed from the Play Store, which the exception of AdAway which can be side-loaded.
Boot loader is version 1.20 rather than the newer 1.22.
All other tweaks and modifications are the same.

Some people have reported that WiFi is more reliable when using a firmware based on 20121120. Due to the size of the custom firmwares and the bandwidth they consume, custom versions of my firmware based on 20121120 are only available to donors.

Feedback

Your feedback is welcome, please use the comments are below.

Headless Debian 6.0 Torrent Server

martin@wimpress.com (Martin Wimpress) — Thu, 21 Mar 2013 17:35:22 +0000

I recently switched ISPs at home and now have unlimited high speed broadband.

Finally I can participate in torrenting Linux .ISO images. I always download the latest distros using BitTorrent and can now contribute to the community by seeding the distros I’ve downloaded.

I have a small (in size and resources) Debian 6.0 headless server at home that I wanted to turn into a torrent box. I’m a big fan of Transmission since it can be managed from the shell, web and Android phone/tablet. Sadly, the Transmission packages in the official Debain squeeze repositories are quite old, 2.03 at the time of writing, and there are no Transmission packages in Debian Backports.

However after flexing my google-fu I found a 3rd party Debian Squeeze repository that includes fairly current (2.73 at the time of writing) Transmission packages specifically for headless use. Yah!

Install Transmission Daemon

First become root.

sudo -s -H

Add the repository key.

apt-key adv --keyserver keyserver.ubuntu.com --recv-key 92B84A1E

Add the repository.

echo "deb http://apt.balocco.name squeeze main" > /etc/apt/sources.list.d/balocco.list

Update the package list.

apt-get update

Install Transmission.

apt-get install transmission-cli transmission-daemon transmission-webinterface

Basic Configuration

The Transmission settings can be found in /etc/transmission-daemon/settings.json.

If transmission-daemon is running when you make changes to settings.json the changes you make will be discarded the next time transmission-daemon is started.

Therefore either stop transmission-daemon before you make any changes or you can make the daemon reload settings.json by sending it the SIGHUP signal.

Connect from anywhere

If you want to be able to connect to Transmission from anywhere on the Internet stop transmission-daemon, make the following changes to settings.json and then start transmission-daemon.

"rpc-password": "YourPlainTextPassword",
"rpc-username": "YourUsername",
"rpc-whitelist-enabled": false,

The rpc-username field will need adding but you can edit the existing entry for rpc-password. Enter the rpc-password as a plain text string, Transmission will automatically convert the password to a hash the next time it is started.

Connect via a browser

You should now be able to access the Transmission web interface via http://yourhost.example.org:9091. If you didn’t change the username and password (you really should) the defaults are:

Username : transmission
Password : transmission

Connect via Android

I have an Android phone and an Android tablet. I use Remote Transmission on my Android devices to manage my torrent box.

Connect via the shell

If, like me, you spend the majority of you time at the shell. Then transmission-remote-cli is probably for you. All my workstation run Arch Linux so I install transmission-remote-cli as follows.

sudo pacman -S transmission-remote-cli

See the GitHub project page for tramission-remote-cli for instructions on how to connect to a remote Transmission daemon.

Block List

Regardless of how you intend to use Transmission you should enable a block list, this can be done via settings.json and the web interface. The following block lists are a good start.

That covers the basics for getting Transmission running on headless Debian 6.0 and how to connect to it from just about anywhere and on any device. I recommend reading the Trasmission Wiki as Transmission is capable of so much more than I have covered in this blog post.

Happy torrenting.

References

Install nginx on Debian

martin@wimpress.com (Martin Wimpress) — Thu, 14 Mar 2013 18:09:21 +0000

My webserver of choice is nginx, it’s resource friendly, fast, reliable and versatile.

I have a resource constrained Debian 6.0 “server” and wanted to deploy nginx on it for testing. Sadly, the nginx package in the Squeeze repositories is very old. Fortunately, the nginx team maintain a Debian package repository.

Add the nginx repository.

sudo nano /etc/apt/sources.list.d/nginx.list

Squeeze

deb http://nginx.org/packages/debian/ squeeze nginx
deb-src http://nginx.org/packages/debian/ squeeze nginx

Wheezy

deb http://nginx.org/packages/debian/ wheezy nginx
deb-src http://nginx.org/packages/debian/ wheezy nginx

Download the nginx package signing key.

wget http://nginx.org/keys/nginx_signing.key

Add the nginx package signing key to the keyring.

sudo apt-key add nginx_signing.key

Update the repositories.

sudo apt-get update

Install nginx.

sudo apt-get install nginx

I run ufw on my VPS so use the following to allow external access to my website.

sudo ufw allow 80/tcp

nginx is installed and can be configured in the usual way.

References

nullmailer on Arch Linux

martin@wimpress.com (Martin Wimpress) — Sat, 09 Mar 2013 23:27:43 +0000

I’ve been a fan of nullmailer for some years now, so much so that I took ownership of the nullmailer package for Arch Linux.

nullmailer is a sendmail/qmail/etc replacement MTA for hosts which relay to a fixed set of smart relays. It is designed to be simple to configure, secure, and easily extendable.

The other advantage nullmailer has compared to similar tools is that is queues email until it is able to deliver it upstream.

Install nullmailer as follows.

packer -S --noedit --noconfirm nullmailer

Configuring nullmailer to relay via Gmail can be achieved using SMTPS or MSA. nullmailer has had these capabilities since 1.10. The following provides some useful clues /usr/lib/nullmailer/smtp --help.

While these examples are specific to relaying via Gmail, you can see it is trivial to adapt them to any other mail host.

Relay via Gmail using MSA

Add to following to /etc/nullmailer/remotes. I prefer this technique.

smtp.gmail.com smtp --port=587 --auth-login --user=you@gmail.com --pass=Yourpassword --starttls

Relay via Gmail using SMTPS

Add to following to /etc/nullmailer/remotes.

smtp.gmail.com smtp --port=465 --auth-login --user=you@gmail.com --pass=Yourpassword --ssl

Once you’ve got /etc/nullmailer/remotes configured start the nullmailer service.

sudo systemctl start nullmailer

To test nullmailer can relay email correctly do the following.

echo "Test 1" | mailx -s "Test One" me@example.org

You can see what nullmailer is up to by checking the systemd journal or syslog (if you’ve syslog enabled systemd). This is how to get the logs from the systemd journal.

journalctl _SYSTEMD_UNIT=nullmailer.service

Or via syslog.

sudo grep nullmailer /var/log/daemon.log

When you’re happy nullmailer is working enable the systemd unit.

sudo systemctl enable nullmailer

Email will now flow as required.

Uncomplicated Firewall (UFW) on Arch Linux

martin@wimpress.com (Martin Wimpress) — Sat, 09 Mar 2013 08:37:43 +0000

While migrating one of my VPS servers to Arch Linux I deployed Uncomplicated Firewall (UFW) to handle basic firewall duties. I like ufw as it provides simple host-based firewall management and, in my opinion, one of the better projects to come out of the Ubuntu camp.

Install ufw as follows.

sudo pacman -Syy -noconfirm --needed ufw

Configuring ufw is simple but make sure you have console access to the host you are configuring just in case you lock yourself out.

ufw allow ssh/tcp

…before running sudo ufw enable. Once the firewall is enabled, adding and removing rules will not flush the firewall, although modifying an existing rule will.

Set the default behaviour to deny all incoming connections.

sudo ufw default deny

sudo ufw limit tcp/22

I’m hosting a few websites on my VPS so I open http and https.

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Enable the ufw systemd unit.

sudo systemctl enable ufw
sudo systemctl start ufw

However, ufw is not enabled at this point. To enable the firewall you also have to do the following.

sudo ufw enable

You can see the status of the firewall using sudo ufw status.

On low-end servers it might be beneficial to disable logging.

sudo ufw logging off

At this point you should have a basic firewall configured and ufw help or the references below will assist you.

References

Installing Nikola on Arch Linux

martin@wimpress.com (Martin Wimpress) — Fri, 08 Mar 2013 17:13:43 +0000

I’ve decided to migrate one of my servers to Arch Linux. I’m not sure that a rolling release distro really suits servers but I’ve enjoyed using Arch Linux over the last year on my workstations and the only way to assess it’s suitability on a server is to try it. So, I’ve decided to migrate my blog to an Arch Linux server.

This blog post describes how to install Nikola on Arch Linux. Nikola is a static site and blog generator written in Python that I’ve been using for a few months.

First you’ll need Python and virtualenvwrapper so read my Python and virtualenv on Arch Linux and Ubuntu blog post and get yourself equipped.

Install the Nikola dependencies.

sudo pacman -S --noconfirm --needed freetype2 libxslt libxml2
sudo packer -S --noconfirm --noedit libjpeg6

Create a virtualenv for Nikola.

mkvirtualenv -p /usr/bin/python2.7 nikola-640

You will notice your shell prompt has changed to indicate that the nikola-640 virtualenv is now active. Install Nikola and the optional libraries I use.

pip install https://github.com/getnikola/nikola/archive/v6.4.0.zip

If you intend to use the Nikola planetoid (Planet generator) plugin you’ll also need to following.

pip install peewee feedparser

Nikola is now installed. nikola help and the Nikola Handbook will assist you from here on.

Dropbox is my Nikola publish button

martin@wimpress.com (Martin Wimpress) — Wed, 06 Mar 2013 19:01:21 +0000

When I migrated my site to Nikola I wanted to ensure I could manage my blog from the shell, the web, Android smartphone or Android tablet. I took some inspiration from Joe Hewitt’s article Dropbox is my publish button and created a free Dropbox account which links to a shared folder on my Dropbox Pro account. I created a simple shell script (invoked via cron every minute) that looks for a trigger file, if the trigger file exists Nikola publishes and deploys the site.

I am able to edit content from anywhere, on any device, and trigger publishing. Very happy.

What follows is how I install Dropbox on headless servers running Arch Linux and Debian/Ubuntu.

Installing Dropbox daemon - all distros

Download the latest Dropbox stable release for 32-bit or 64-bit.

wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86"
wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86_64"

Extract the archive and install Dropbox in /opt.

cd
tar -xvzf dropbox.tar.gz
sudo mv ~/.dropbox-dist /opt/dropbox
sudo find /opt/dropbox/ -type f -exec chmod 644 {} \;
sudo chmod 755 /opt/dropbox/dropboxd
sudo chmod 755 /opt/dropbox/dropbox
sudo ln -s /opt/dropbox/dropboxd /usr/local/bin/dropboxd

Run dropboxd.

/usr/local/bin/dropboxd

You should see output like this:

This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?host_id=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx to link this machine.

Visit the URL, login with your Dropbox account and link the account. You should see the following.

Client successfully linked, Welcome Web!

dropboxd will now create a ~/Dropbox folder and start synchronizing. Stop dropboxd with CTRL+C.

Arch Linux - systemd

Run Dropbox as daemon with systemd. Create /usr/lib/systemd/system/dropbox@.service with the following content.

[Unit]
Description=Dropbox
After=local-fs.target network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/dropboxd
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
User=%I

[Install]
WantedBy=multi-user.target

Enable the daemon for your user, run the following replace<user> with your username. This will ensure Dropbox is started when the system boots.

sudo systemctl enable dropbox@<user>
sudo systemctl start dropbox@<user>

Debian/Ubuntu - init.d

Run Dropbox as daemon with init.d. Create /etc/init.d/dropbox with the following content, replacing <user> with your username.

#!/bin/sh
### BEGIN INIT INFO
# Provides: dropbox
# Required-Start: $local_fs $remote_fs $network $syslog $named
# Required-Stop: $local_fs $remote_fs $network $syslog $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# X-Interactive: false
# Short-Description: dropbox service
### END INIT INFO

DROPBOX_USERS="<user>"
DAEMON=/opt/dropbox/dropbox

start() {
 echo "Starting dropbox..."
 for dbuser in $DROPBOX_USERS; do
 HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
 if [ -x $HOMEDIR/$DAEMON ]; then
 HOME="$HOMEDIR" start-stop-daemon -b -o -c $dbuser -S -u $dbuser -x $HOMEDIR/$DAEMON
 fi
 done
}

stop() {
 echo "Stopping dropbox..."
 for dbuser in $DROPBOX_USERS; do
 HOMEDIR=`getent passwd $dbuser | cut -d: -f6`
 if [ -x $HOMEDIR/$DAEMON ]; then
 start-stop-daemon -o -c $dbuser -K -u $dbuser -x $HOMEDIR/$DAEMON
 fi
 done
}

status() {
 for dbuser in $DROPBOX_USERS; do
 dbpid=`pgrep -u $dbuser dropbox`
 if [ -z $dbpid ] ; then
 echo "dropboxd for USER $dbuser: not running."
 else
 echo "dropboxd for USER $dbuser: running (pid $dbpid)"
 fi
 done
}

case "$1" in
 start)
 start
 ;;
 stop)
 stop
 ;;
 restart|reload|force-reload)
 stop
 start
 ;;
 status)
 status
 ;;
 *)
 echo "Usage: /etc/init.d/dropbox {start|stop|reload|force-reload|restart|status}"
 exit 1
esac

exit 0

Enable the init.d script.

sudo chmod +x /etc/init.d/dropbox
sudo update-rc.d dropbox defaults

Install Dropbox client - all distros

It is recommended to download the official Dropbox client to configure Dropbox and get its status.

wget "http://www.dropbox.com/download?dl=packages/dropbox.py" -O dropbox-cli
chmod 755 dropbox-cli
sed -i s'/#!\/usr\/bin\/python/#!\/usr\/bin\/env python2/' dropbox-cli
sudo mv dropbox-cli /usr/local/bin/

For usage instructions run dropbox-cli help.

Disable LAN Sync

Stop Dropbox from sending LAN Sync broadcasts every 30 seconds over port 17500.

dropbox-cli lansync n

I’m planning to make more use of Dropbox for content management and content delivery, blog posts to follow.

References

Microsoft Office 2010 on Arch Linux and Ubuntu

martin@wimpress.com (Martin Wimpress) — Tue, 29 Jan 2013 16:16:47 +0000

We have a mix of Linux and Windows users at work. My department use Linux and the rest of the business use Windows. We been running a mixture of LibreOffice and Microsoft Office, which works pretty well until you start trying to collaborate, then it gets messy pretty quickly.

So, it was decided at the end of 2012 to migrate everyone, including the Linux users, to Microsoft Office 2010.

What follows is an installation guide for Wine and the 60 day trial version of Office Home and Business 2010 on Arch Linux and Ubuntu. Most of this information was sourced from the Wine AppDB

http://appdb.winehq.org/objectManager.php?sClass=version&iId=17336

Wine for Ubuntu 12.04 LTS (or better)

Install Wine on Ubuntu as follows.

sudo apt-add-repository ppa:ubuntu-wine/ppa
sudo apt-get update
sudo apt-get install ttf-mscorefonts-installer samba wine1.5 wine-gecko1.8 wine-mono

For 64-bit also install the following.

sudo apt-get install ia32-libs

Wine for Arch Linux

Update I updated this section on August 8th 2013 to reflect recent changes in Arch Linux

Install Wine on Arch Linux as follows.

sudo pacman -S --needed icoutils libwbclient libxslt lib32-mpg123 p11-kit lib32-p11-kit samba wine winetricks wine-mono wine_gecko
sudo packer -S --noedit --noconfirm ttf-ms-fonts

For 64-bit also install the following.

sudo packer -S --noedit --noconfirm lib32-libwbclient lib32-libxslt

Installing Office 2010

Once Wine is installed, installing Office 2010 is the same for Arch Linux and Ubuntu.

Create a clean wine prefix.

export WINEPREFIX="${HOME}/.msoffice2010"
export WINEARCH="win32"
winecfg

Click the Libraries tab, select riched20 and click Add. The default entry should read riched20 (native, builtin). Click Apply, then click OK. This will ensure that PowerPoint starts and selection boxes display correctly.

Install libxml6 and corefonts with winetricks

winetricks msxml6 corefonts

Start the Office 2010 setup. In the example below X17-75058.exe is the name of the 60 day trial version of Office Home and Business 2010 that I downloaded.

wine X17-75058.exe

Follow the installation wizard, we are only interested in running the essentials, Word, Excel and PowerPoint. This is what I selected during the install.

Enter your serial number.
Leave ticked Attempt to automatically activate my product online.
Click Continue
Tick I accept the terms of this agreement
Click Continue
Click Customise
- Microsoft Access (Trial) [Not Available]
- Microsoft Excel [Run all from My Computer]
- Microsoft OneNote [Not Available]
- Microsoft Outlook [Not Available]
- Microsoft PowerPoint [Run all from My Computer]
- Microsoft Publisher (Trial) [Not Available]
- Microsoft Visio Viewer [Run from My Computer]
- Microsoft Word [Run all from My Computer]
- Office Shared Features [Defaults]
- Office Tools [Defaults]
Click Instal Now.
Click Close.

That’s it. Office 2010 is installed and should be associated with the appropriate file types.

Some Issues

Here are some of the issues we noticed running Office 2010 under Wine.

Always install Office 2010 into it’s own WINEPREFIX. You are less likely to run into problem that way.
Online updates do not work. Fortunately, the trial installer has SP1 integrated.
If you purchase Office 2010 licenses you can still use the trial installer with your purchased license key(s).
We did test a trial of CrossOver. However, it wouldn’t activate Office 2010 on Arch Linux but did activate on Ubuntu.
Files saved to cifs mounts are set read-only. This might be a Wine issue or due to the unusual way we have our file server configured, we are still investigating.

Uninstalling Office 2010

Should you ever need to, you can uninstall Office 2010 as follows.

rm -rfv ${HOME}/.msoffice2010/
rm -rfv ~/.local/share/applications/wine-extension-*
rm -rfv ~/.local/share/applications/wine/Programs/Microsoft\ Office/

The nano text editor

martin@wimpress.com (Martin Wimpress) — Tue, 22 Jan 2013 15:25:47 +0000

I use the nano text editor in preference to vim and have done for years. This is because we used Pine for email at university and my first job, the Pico text editor was used to compose mail messages. Due to the binary only distribution of pico, nano was created as an free software alternative. And that is why I use nano.

Since I migrated my blog to Nikola I’m using nano more frequently as I typically write my blog posts on a remote shell, so I thought I’d spend some time to tweak nano a little.

Keybindings

I refreshed my memory of some of the keyboard shortcuts available in nano to be a little more efficient.

Syntax Highlighting

Syntax Highlighting is the killer feature for nano that I’ve never bothered to configured in the past. I based my configuration one those provided by Craig Barnes. He uses mixins to ensure a consistent colour theme for all the language highlighters. I don’t use his custom key bindings however, it gets confusing when connecting to different hosts that have a default nano configuration.

https://github.com/craigbarnes/nanorc

Python and virtualenv on Arch Linux and Ubuntu

martin@wimpress.com (Martin Wimpress) — Tue, 11 Dec 2012 20:38:50 +0000

We use Python for pretty much all our software development at work. We also use virtualenv and virtualenvwrapper extensively, both for development and deployment.

Why is virtualenv so great?

It just is. Read the virtualenv documentation. If you’re a Python developer you need virtualenv in your life. You also need virtualenvwrapper too.

virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool for creating isolated Python development environments.

Installing Python and virtualenvwrapper

Outlined below is how I install Python and virtualenvwrapper. We have not yet made the jump to Python 3 at work, hence the references to Python 2.6 and 2.7. Some of us develop on Arch Linux, but all deployments are on Ubuntu.

Arch Linux

As Arch Linux is a rolling release we can simply install everything via pacman.

sudo pacman -Syy
sudo pacman -S --needed --noconfirm python-pip python-setuptools python-virtualenv
sudo pacman -S --needed --noconfirm python2-pip python2-setuptools python2-virtualenv python-virtualenvwrapper"

Simple.

Ubuntu

The following was done on Ubuntu Lucid 10.04 LTS.

Add some essential PPAs.

sudo apt-add-repository ppa:bzr/ppa
sudo apt-add-repository ppa:git-core/ppa
sudo apt-add-repository ppa:fkrull/deadsnakes

Update the system and install Python 2.6 and 2.7.

sudo apt-get update
sudo apt-get install libpython2.6 python2.6 python2.6-dev python2.6-minimal
sudo apt-get install libpython2.7 python2.7 python2.7-dev python2.7-minimal

Remove any apt installed Python packages that we are about to repalce. The versions of these packages in the Ubuntu repos and PPAs are too old.

sudo apt-get purge python-setuptools python-virtualenv python-pip python-profiler

Install distribute.

curl -O http://python-distribute.org/distribute_setup.py
sudo python2.6 distribute_setup.py
sudo python2.7 distribute_setup.py

Install pip.

curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
sudo python2.6 get-pip.py
sudo python2.7 get-pip.py

Use pip to install virtualenv and virtualenv wrapper.

sudo pip-2.6 install virtualenv --upgrade
sudo pip-2.7 install virtualenv --upgrade
sudo pip install virtualenvwrapper

Fairly simple.

The Snakepit

This step is common to Arch Linux and Ubuntu. Create a “Snakepit” directory for storing all the virtualenvs.

mkdir ~/Snakepit

Add the following your ~/.bashrc to enable virtualenvwrapper.

export WORKON_HOME=${HOME}/Snakepit
if [ -f /usr/local/bin/virtualenvwrapper.sh ]; then
 source /usr/local/bin/virtualenvwrapper.sh
elif [ -f /usr/bin/virtualenvwrapper.sh ]; then
 source /usr/bin/virtualenvwrapper.sh
fi

Creating a virtualenv

Open a new shell to ensure that the virtualenvwrapper configuration is active.

The following will create a new virtualenv called Nikola5 based on Python 2.7 and will not give access to the global site-packages directory.

mkvirtualenv -p python2.7 --no-site-packages ~/Snakepit/Nikola5

mkvirtualenv_help shows a full list of arguments, the -r switch can install all the packages listed in a pip requirements file into the newly created virtualenv. Very useful.

Working on a virtualenv

To workon, or activate, an existing virtualenv do the following.

workon Nikola5

You can switch to another virtualenv at any time, just use workon envname. Your shell prompt will change while a virtualenv is being worked on to indicate which virtualenv is currently active.

When you are finished working in a virtualenv you can deactivate it by simply executing:

deactivate

That just about sums up my notes.

SSH brute force defense

martin@wimpress.com (Martin Wimpress) — Mon, 26 Nov 2012 13:14:09 +0000

I have several VPS hosts with different providers using different virtualisation platforms. Naturally I have OpenSSH running on these VPS hosts and deploy either DenyHosts or Fail2Ban to add an extra security layer to thwart SSH brute force attacks and other abuse.

DenyHosts blocks brute force attacks by adding offending IP addresses to /etc/hosts.deny. It therefore requires the SSH server is configured with tcp_wrappers. Arch Linux dropped support for tcp_wrappers so DenyHosts is not suitable for Arch. Fail2Ban supports blocking via iptables and/or tcp_wrappers and can also block offending hosts that are abusing services other than just sshd.

DenyHosts on Ubuntu

Here is a simple example for DenyHosts on Ubuntu Lucid 10.04 LTS Server.

sudo apt-get install denyhosts

That’s it. The default configuration will provide suitable prevention, but do take a look at /etc/denyhosts.conf for a full run down of all available options. I use the defaults with the following exceptions:

PURGE_DENY = 5d
PURGE_THRESHOLD = 2
ADMIN_EMAIL =
SYSLOG_REPORT=NO

You might want to consider whitelisting some of your own IP address. Create a file called allowed-hosts in /var/lib/denyhosts and list each of your “trusted” IP addresses.

DenyHosts can be restarted by executing:

sudo /etc/init.d/denyhosts restart

Fail2Ban on Arch Linux

Fail2Ban now supports systemd.

Configuration files are stored in /etc/fail2ban. General configuration is /etc/fail2ban/jail.conf, but this file might be overwritten in the future. To preserve customisations, create /etc/fail2ban/jail.local and add your local configuration settings to it. In the example below some IP addresses are whitelisted and the default backend is set to systemd:

[DEFAULT]

ignoreip = 172.16.0.2/32
backend = systemd

Next create a custom sshd configuration in /etc/fail2ban/jail.d/sshd.conf which will temporarily ban offending IP addresses.

# fail2ban SSH
# block ssh after 3 unsuccessful login attempts for 10 minutes
[sshd]
enabled = true
action = iptables[chain=INPUT, protocol=tcp, port=22, name=sshd]
maxRetry = 3
findtime = 600
bantime = 600
port = 22

The ‘action’ creates DROP rule in iptables after 3 unsuccessful login attempts, valid for 10 minutes (bantime). Findtime defines time frame in which fail2ban will count failed login attempts from logs, so if one IP has 3 incorrect login attempts in last 10 minutes, it will be banned.

Enable and start the Fail2Ban daemon.

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Basic commands for fail2ban-client:

fail2ban-client start sshd
fail2ban-client stop sshd
fail2ban-client reload sshd
fail2ban-client status sshd
fail2ban-client set sshd unbanip 172.16.0.4

See the Arch Linux Fail2Ban Wiki page for more details.

References

http://krisko210.blogspot.co.uk/2014/03/setting-up-fail2ban.html

SSH best practice

DenyHosts and Fail2Ban do not provide complete protection against SSH brute force attacks. I employ other SSH best practice to better secure the SSH services I expose to the Internet, and so should you. The following is a good reference.

Getting started with SSH security and configuration

Other SSH brute force prevention tools

In the interests of fairness, other SSH brute force preventation tools are available.

Do you know any other tools that help prevent SSH brute force attacks?

Netflix for ArchLinux

martin@wimpress.com (Martin Wimpress) — Thu, 22 Nov 2012 22:14:47 +0000

Something wonderful has happened! Jesus Alvarez has created pre-compiled packages for netflix-desktop for Arch Linux.

There is no need for me to repeat the installation instructions as they are clearly documented on the page below.

Arch Netflix - Netflix on the desktop through WINE

The source code for both of the packages can be found at archnetflix-github.

For reporting problems, please see the AUR page at netflix-desktop-aur or the announcement forum posting.

Service Monitoring with StatusCake

martin@wimpress.com (Martin Wimpress) — Tue, 20 Nov 2012 17:14:47 +0000

We use two server monitoring services at work, one that does external availability checks of critical services and another agent based monitor that reports various system metrics. Neither of these monitoring tools offer sensible free plans.

I recently setup monitor.us to do external monitoring of my VPS hosts. It’s very good and offers many tests (internal and external), smartphone apps and reports. All are available on the free plan. However, the user interface is cumbersome, the reports are ugly and it feels too formal for my personal use. Therefore I no longer use it, but it does a job, so give it a try.

I don’t require agent based systems monitoring. Simple external availability monitoring is all I need for the few personal websites and services I run. So I went off to find and alternative and after some google-fu I found StatusCake.

StatusCake is easy to configure, offers monitoring of http(s), TCP ports and ICMP Ping. Notifications can be delivered via Boxcar, Pushover, NotiApp, Skype, Twitter, email and SMS. StatusCake also looks beautiful and their Real Browser Testing feature is very interesting.

Several plans are available at attractive prices and the free plan supports unlimited sites. Brilliant! Perfect for personal use. StatusCake has only been around since August 2012 (as far as I can tell) but is already shaping up to be real contender to Pingdom.

What monitoring services are you using?

GitHub Analytics

martin@wimpress.com (Martin Wimpress) — Fri, 26 Oct 2012 16:39:00 +0000

As I mentioned in a previous post we are prepairing to migrate our Bazaar repositories to Git, or more precisely to GitHub. This migration also heralds the Open Source releases of many of the core technologies we’ve been developing at Flight Data Services for the last few years.

I want to track visits for our GitHub projects. A bit of Googling turned up githalytics which enables you to track visits and page views for your GitHub projects using [http://www.google.com/analytics/](Google Analytics).

To use it, create a new Google Analytics property ID for your GitHub project, head over to http://githalytics.com/ and complete the web form. You’ll be provided a Markdown snippet to insert in your projects README.md. It will look something like this:

[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f "githalytics.com")](http://githalytics.com/YourGitHubName/YourGitHubProject)

Add the snippet to README.md and push the changes. When someone visits your GitHub project page, the visit will be tracked. Great!

However, we write all our documentation using reStructuredText. But after a quick Twitter and email exchange with Dimitrios from githalytics and I had a reStructuredText snippet. It looks something like this:

image:: https://cruel-carlota.pagodabox.com/0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f0f
 :alt: githalytics.com
 :target: http://githalytics.com/YourGitHubName/YourGitHubProject

Same drill, except add this snippet to README.rst and push the changes.

So there you have it, Google Analytics tracking of your GitHub project landing page. If you have more than one project, create a Google Analytics property ID and githalytics tracking snippet for each project.

References

Migrating Bazaar to Git

martin@wimpress.com (Martin Wimpress) — Tue, 23 Oct 2012 11:50:00 +0000

We have been using Bazaar for source control at work for nearly five years. We are about to Open Source most of our core technologies and decided that GitHub is the best way to encourage community participation. We have signed up for a Silver plan at GitHub and will migrate all our Bazaar repositories to Git.

I have a few personal projects in Bazaar repositories hosted on Launchpad. I decided to migrate my projects to GitHub in order to learn the migration process. What follows is an overview of how I did it using a fresh virtual machine running Ubuntu 10.04 LTS Server. I used a little project of mine called nullserv in the examples below.

Add the Bazaar and Git PPAs.

sudo apt-get install python-software-properties
sudo apt-add-repository ppa:bzr/ppa
sudo apt-add-repository ppa:git-core/ppa
sudo apt-get update

Install bzr (and its requirements), curl and git.

sudo apt-get install bzr bzr-fastimport curl git python-paramiko

Add the SSH keys and identify yourself.

bzr whoami "Your Name <name@example.org>"
git config --global user.name "Your Name"
git config --global user.email you@example.org

If your Bazaar repository is hosted on Launchpad assert your identity.

bzr launchpad-login flexiondotorg

Branch the Bazaar repository.

bzr branch lp:~flexiondotorg/+junk/nullserv
cd nullserv
git init
bzr fast-export --plain `pwd` | git fast-import

This step is optional. It will delete and commit the deletions to the Bazaar repository.

for FILE in *; do rm -rfv "${FILE}"; done
echo "This repository has been migrated to Git. https://github.com/flexiondotorg/nullserv" > README
bzr add README
bzr commit -m "This repository has been migrated to Git. https://github.com/flexiondotorg/nullserv"
bzr push :parent

Remove the Bazaar repository and reset the Git repository.

rm -rf .bzr README
git reset HEAD

Create .gitattributes to normalise line endings.

cat >.gitattributes<<ENDGITATTRIBS
# Normalise line endings:
* text=auto

# Prevent certain files from being exported:
.gitattributes export-ignore
.gitignore export-ignore
ENDGITATTRIBS

git add .gitattributes

Migrate .bzrignore to .gitignore.

git mv .bzrignore .gitignore
echo >> .gitignore
echo "# Keep empty directories:" >> .gitignore
echo "!*/.git*" >> .gitignore

Ensure empty directories are retained by git.

find -empty -type d -not -iwholename '*.git*' -exec touch '{}/.gitkeep' ";"
git add **/.gitkeep

Commit the migrated repository

git commit -a -m "Migrated from Bazaar to Git."

Thanks to Chris for pointing out git filter-branch in the comments. If you need to modify the author info in your repository history, you can do so with this, just replace the names and email addresses accordingly.

BEWARE! This should not be performed on a repo that has been shared with others. Use at your own risk.

git filter-branch --commit-filter '
 if [ "$GIT_COMMITTER_NAME" = "Fred" ]; then
 GIT_COMMITTER_NAME="Fred Flintstone";
 GIT_AUTHOR_NAME="Fred Flintstone";
 GIT_COMMITTER_EMAIL="fred@example.org";
 GIT_AUTHOR_EMAIL="fred@example.org";
 git commit-tree "$@";
 elif [ "$GIT_COMMITTER_NAME" = "Barney" ]; then
 GIT_COMMITTER_NAME="Barney Rubble";
 GIT_AUTHOR_NAME="Barney Rubble";
 GIT_COMMITTER_EMAIL="barney@example.org";
 GIT_AUTHOR_EMAIL="barney@example.org";
 git commit-tree "$@";
 else
 git commit-tree "$@";
 fi' HEAD

If you want to delete any files from the commit history, you can optionally do that now.

git filter-branch -f --index-filter "git rm --cached --ignore-unmatch *.THIS *.THAT" \
--prune-empty --tag-name-filter cat -- --all
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now

Resume here, regardless of whether you deleted any files from the commit history or not. Remove everything from the index.

git rm --cached -r .

Write both the index and working directory from git’s database.

git reset --hard

Prepare to make a commit by staging all the files that will get normalized. This is your chance to inspect which files were never normalized. You may get lots of messages like: warning: CRLF will be replaced by LF in file.

git add .
git commit -m "Forced line endings to eol=lf"

Aggressively pack the repository.

git gc --aggressive --prune=now

At this point you have a migrated git repository. You can poke about a check that everything is present and correct.

Optionally you can create a new GitHub repository using their API. Replace USER and PASS with your GitHub login crednetials.

curl -u 'USER:PASS' https://api.github.com/user/repos -d '{"name":"nullserv"}'

If you want to create repositories for an Organisation the following will work. Replace YourOrganisation with the organisation name your are a member of.

curl -u 'USER:PASS' https://api.github.com/orgs/YourOrganisation/repos -d '{"name":"nullserv"}'

Private repositories can be created, providing you have a paid GitHub account, by changing the POST data as follows.

'{"name":"nullserv","private":"true"}'

Lastly, push to the newly created GitHub repo.

git remote add origin git@github.com:flexiondotorg/nullserv.git
git push -u origin master

All done, the Bazaar repository has been crippled and the Git repository is ready for use on GitHub.

References

Migrating Wordpress to Nikola

martin@wimpress.com (Martin Wimpress) — Mon, 22 Oct 2012 17:13:43 +0000

I recently migrated three sites from a self hosted Wordpress installation to Nikola. Nikola is a static site and blog generator written in Python.

Although I use both reStructuredText and Markdown, I decided to migrate my Wordpress content to Markdown.

This is by no means an exhaustive Wordpress to Nikola migration guide but it should provide enough clues for anyone else wanting to do the same. The following was done on Ubuntu 10.04 LTS.

Export the Wordpress content. Tools -> Export -> All Content

Use xmllint to find any errors in the Wordpress XML export and fix them.

Nikola 5

UPDATE! I’ve added the instructions for install Nikola 5 since fist publishing this post.

Install Nikola 5 in a virtualenv using virtualenvwrapper.

cd ~
sudo apt-get install libxslt1-dev libxml2-dev libjpeg62-dev python2.6-dev
wget http://nikola-generator.googlecode.com/files/nikola-5.zip
unzip ~/nikola-5.zip
mkvirtualenv -i markdown -r ~/nikola-5/requirements.txt --use-distribute nikola-5
cd ~/nikola-5
python setup.py install

Nikola 4

Install Nikola 4.0.3, in a virtualenv using virtualenvwrapper. Nikola 4.0.3 spits deprecation warnings with doit>=0.16.1 hence the use of sed.

cd ~
sudo apt-get install libxslt1-dev libxml2-dev libjpeg62-dev python2.6-dev
wget http://nikola-generator.googlecode.com/files/nikola-4.0.3.zip
unzip ~/nikola-4.0.3.zip
sed -i 's/>=0\.16/==0\.16/' ~/nikola-4.0.3/requirements.txt
mkvirtualenv -i markdown -r ~/nikola-4.0.3/requirements.txt --use-distribute nikola
cd nikola-4.0.3
python setup.py install

Import the Wordpress content.

cd
nikola import_wordpress wordpress.linted.xml

Use html2text to convert the HTML markup in new_site/posts/*.wp to real Markdown.

Use the Disqus Wordpress Plug-in to migrate Wordpress comments to Disqus.

If required, generate a list of the Wordpress URLs for Nikola redirections.

grep "<link>" wordpress.linted.xml | sed -e 's/<link>//g' -e 's/<\/link>//g'

I migrated from several sub-domains to one top-level and the Wordpress URLs I was using can’t be persevered with Nikola. I use a combination of Nikola redirects and nginx configuration to handle the re-directions.

At this point the bulk of the migration is done. I tweaked the Nikola conf.py to use .md files instead of .wp, added some assets to the Nikola files directory, configured deployments and updated the theme. I also decided to axe some obsolete blog posts.

Migrating to Disqus has been very frustrating and although my comments have now been migrated the Migrate Threads has yet reflect the new URLs of my posts. There is no visibility of what, if anything, is happening when you execute the Disqus URL Mapper. This is not a Nikola issue.

I am extremely happy with Nikola itself and it has proved itself flexible and I can now capture my notes in a familiar format and in a familiar environment, Python. Next steps are to integrate Nikola with Dropbox so I can publish from any device with ease and add a search facility.

draw.io - Draw diagrams online, free

martin@wimpress.com (Martin Wimpress) — Wed, 17 Oct 2012 13:11:56 +0000

Yesterday I needed to create a data flow diagram for one of our technology partners at work. They are presenting to the FAA and need to explain how we (Flight Data Services) can replay the data from tens of thousands of flights in extremely short order.

Anyway, I went looking for a good tool to draw the diagram. I’ve tried several in the past and never really been satisfied. I found draw.io and it is really good. This is how they describe it.

draw.io is an online diagramming application and github project. It features the full range of visual configuration you expect, as well as web application features such as a full range of export options, a large collection of icons, real-time collaboration and embedded widget sharing.

The shape library is extensive and pretty consistent. I was able to export the diagram in the formats I wanted. Overall, it’s not half bad and completely free.

Quality Android tablet on a budget - Ployer Momo8 IPS

martin@wimpress.com (Martin Wimpress) — Thu, 27 Sep 2012 20:14:43 +0000

I’ve recently taken delivery of my first Android tablet, the Ployer Momo8 IPS.

I purchased mine from GTR Electronics as they have a track record for excellent customer service, provide 12 month warranty and their devices are free from Chinese apps and bloatware. I wrote up a review on Amazon.co.uk, see below.

A quality Android tablet with a budget price

As I mention in the review above, I’ve rooted the Momo8 IPS. The following were useful references for the rooting.

Update Since writing this blog post I’ve made a pre-rooted custom firmware for the Ployer Momo 8 IPS. Installing it may be easier than following the instructions below.

Ployer Momo8 IPS Custom Firmware

Here are the basic steps to getting root on the Momo8 IPS.

Enable USB Debugging on the Momo8 IPS - Settings –> Developer Options
Find a Windows computer.
Download and install Momorobo 2.0.2.290.
Start Moborobo and connect the Momo8 IPS to the Windows computer. Wait for Moborobo to establish a connection to the tablet.
Download and install ZhuoDaShi 2.2.9.
Start ZhuoDaShi. It is a Chinese language application only, so this is somewhat tricky.
- Wait for ZhuoDaShi connect to your Momo8 IPS.
- Click the highlighted text that includes the word ROOT among some Chinese text then click on the big ROOT button.
- After a short while you should see some Chinese text in green. Click the large button.
You should now have root and the SuperSU app will be listed in your apps.
Install Root Checker on the Momo8 IPS to ensure you have root.
Once satisfied you have root (I ran Titanium Backup to be absolutely sure) you can uninstall the Momorobo daemon and ZhuoDaShi app the from your Momo8 IPS.
The version of SuperSU installed by ZhuoDaShi is quite old and can’t be updated via the Google Play Store. To remedy this do the following.
- Under SuperSU -> Settings select Switch superuser app
- Now open Google Play Store and install SuperSU.

Hope this is helpful to someone else, but proceed with caution.

Android Gmail Conversation View

martin@wimpress.com (Martin Wimpress) — Tue, 25 Sep 2012 19:38:34 +0000

I recently migrated several domains I manage for family and friends to the “free” version of Google Apps.

I setup the Gmail app on their Android phones for them and email arrives promptly and spam is pretty much eradicated. Happy days.

However, the Gmail app for Android only supports a conversation view of emails. My wife and I hate that and have actually missed emails because new emails get burried in an earlier email conversation way down your inbox. Frustratingly changing the preferences on the Gmail website to disable the conversation view does not propagate to the Gmail app.

The best solution to this that I have found is to to not use the Gmail app and install K-9 Mail instead connected to Google App accounts via IMAP and SMTP. K-9 Mail (only?) supports a traditional list view of your emails and unlike the built in Email app for Android K9-Mail supports push notifications so you get near instantaneous notification of new email.

If you want to support the K-9 Mail developer then consider purchasing Kaiten Mail which is an enhanced version of K-9 Mail and is particularly good on Android tablets.

MATE Desktop

martin@wimpress.com (Martin Wimpress) — Fri, 01 Jun 2012 00:00:00 +0000

The MATE Desktop Environment is the continuation of GNOME 2. MATE Desktop provides an intuitive and attractive desktop environment using traditional metaphors for Linux and other Unix-like operating systems.

Organisation: MATE Desktop
Date: June 2012 - date
Role: Developer & Community Manager

Arch Linux Angel

martin@wimpress.com (Martin Wimpress) — Tue, 15 May 2012 11:12:13 +0000

I’ve been working a shell script for Arch Linux that automatically configures my preferred GNOME 3 setup on my netbook, laptops and workstations. The main features are:

Quickly deploys Arch Linux to my specifications
Supports i686 and x64_64.
Detects ATI/AMD, Intel and Nvidia chipsets and configures the Open Source video drivers and enables early KMS.
Hardware and location aware. Installation and configuration can be different for Home vs Work or Desktop vs Netbook.
Detects and correctly configures some device specific hardware, such as touch screens and wireless drivers.
Automatically configures DAEMONS array.
Includes custom power management hooks for pm-utils.
Designed to safely run multiple times so that it can be used as a tool for keeping all systems consistent.

I’ve dubbed this script Arch Angel. I’m undecided if I’ll release it publicly since it is very much my personal preferences and to some extent my colleagues at work. I suppose the real reason for this post is that I’ve been wanting to take Shelr for a test drive, so click the Play button below to see an example run of Arch Angel.

Take Mark Shuttleworth's advice, try Arch Linux

martin@wimpress.com (Martin Wimpress) — Wed, 02 May 2012 19:00:19 +0000

In a recent Ubuntu community feedback session run on IRC Mark Shuttleworth said:

if you are a super-technologist then there is value in learning all about linux from every angle try arch. try gentoo. try fedora. try debian. try suse. they are all good

So, I did. I tried Arch Linux and it is is not just good, it’s truly brilliant! The truth is, I didn’t take Mark Shuttleworth’s advice. I took the advice of a quietly spoken work colleague. Whenever he saw me getting frustrated with Ubuntu he reminded me (quietly) that he was running Arch Linux and that it was really rather good. It turns out I should have listened to my colleague years ago, because he was right all along. He often is.

I started learning Arch Linux a few weeks ago and as of the time of writing all our home computers are running Arch Linux and so is my office workstation. I moved away from Ubuntu (which I’ve been using daily since 2004) because:

I often find myself needing/wanting updated packages on my workstations. Ubuntu + PPAs just wasn’t doing it for me anymore. I concluded I needed a rolling release distribution.
In my humble opinion, Unity is a software engineering solution developed (by programmers) not designed using UI-patterns (by user interface designers) consequently it sucks the big one.
It was less hassle for me to switch to Arch Linux than bend Ubuntu to my will.

That said, I do wish Canonical every success with Unity, they’ll need it, as it needs some polish and refinement as it stands today. For anyone interested in giving Arch Linux a try here are some observations.

Arch Linux does not hold you hand, you should be competent with a Linux based distribution. I’ve been using Linux since 0.99patch2 so I think I qualify.
Read the Unofficial Beginners’ Guide and Official Installation Guide before you install anything.
Install Arch Linux in a virtual machine to get acquainted.
Be prepared to fix your own problems. Fortunately the Arch Linux Wiki and Arch Linux Forums are a great source of good information.

Here is a list of what I most like about Arch Linux:

It’s a rolling release. I’ve already progressed through several kernels and been upgraded from GNOME 3.2 to 3.4. No fuss, no mess.
It keeps out of my way, I get to build the OS they way I want it. Yes, I’m looking at you Unity, Overlay Scrollbars, Unity App Indicators, Ubuntu One and upstart!
The AUR. It’s like a great big PPA.
Creating my own packages is super simple.
Adding my own packages to the AUR is super simple. I’ve already contributed 8 packages.
I’m learning more about Linux than I have done in years.

Here’s what I’m not so keen about, but can live with.

Some of the comments in the Forums and AUR are from real arse hats with a superiority complex. New comers may be put off by their tone and look elsewhere.
The AUR. Package quality in the AUR is variable, some are pretty poor. So long as the maintainer is not one of the aforementioned arse hats then contributing fixes and improvements is easy.
It’s a rolling release, so may not be suitable for serious server deployments. That said, I will migrate my own servers to Arch Linux in due course and see if I can prove myself wrong.

So there you have it, I like Arch Linux very much and recommend it to other technically competent individuals looking to regain control of their computer(s).

Install Sun Java 6 JRE and JDK from .deb packages

martin@wimpress.com (Martin Wimpress) — Mon, 16 Jan 2012 13:20:05 +0000

Canonical disabled my Java PPA at the end of last week. So I’ve developed another solution for installing Java on Ubuntu which doesn’t infringe any copyrights, licenses, terms of use or CoC’s. However, by running this script to download Java you acknowledge that you have read and accepted the terms of the Oracle end user license agreement.

http://www.oracle.com/technetwork/java/javase/terms/license/

My script is an automated wrapper for Janusz Dziemidowicz Debian packaging scripts for Java 6. My new script simply downloads the Java binary installers from Oracle, builds the .deb packages locally on your computer and creates a local ‘apt’ repository for them. Once my script has been executed you can then ‘apt-get’ install/upgrade Java 6 from your local repository. Packages are compatible with “official” Ubuntu ones and pre-existing Java 6 packages will upgrade cleanly. You can find the script and full usage instructions on github.

Please read the README file for a more detailed explanation of how the script works and how to use it. If anyone has any problems, then please submit a ticket on my Issue Tracker.

Sun Java 1.6.0.30 packages for Ubuntu

martin@wimpress.com (Martin Wimpress) — Tue, 10 Jan 2012 10:53:50 +0000

Update Friday 13th January 2012

My Java PPA has been disabled by Canonical, possibly because they violate the Ubuntu CoC and PPA terms of use, as Jef Spaleta noted in the comments below, although I’ve had no communication from Canonical at this time. I’m preparing an alternative solution, for those of you who need Sun Java 6, that doesn’t violate and copyrights, CoCs or terms of use. A new blog post will be made when that alternate solution is available.

Update Monday 16th January 2012

I’ve developed another solution for installing Java 6u30 on Ubuntu which doesn’t infringe any copyrights, licenses, terms of use or CoC’s.

Install Sun Java 6 JRE and JDK from .deb packages

~~Sun Java 6 packages are being removed from Ubuntu in the near future for the following reasons:~~

~~As of August 24th 2011, Canonical no longer have permission to redistribute new Java packages as Oracle has retired the “Operating System Distributor License for Java”.~~
~~Oracle has published an advisory about security issues in the version of Java currently in the partner archive. Some of these issues are currently being exploited in the wild.~~
~~Due to the severity of the security risk, Canonical released a security update for the Sun JDK browser plugin which disables the plugin on all machines.~~
In the near future, Canonical will remove all Sun JDK packages from the Partner archive. This will be accomplished by pushing empty packages to the archive, so that the Sun JDK will be removed from all users machines when they do a software update. Users of these packages who have not migrated to an alternative solution will experience failures after the package updates have removed Oracle Java from the system.

~~See the full Canonical notice below.~~

~~https://lists.ubuntu.com/archives/ubuntu-security-announce/2011-December/001528.html~~

~~My personal motivations for creating this PPA are as follows:~~

~~I require Sun Java 6 for two enterprise applications we use at work. OpenJDK is not fully compatible.~~
~~I require Sun Java 6 for two desktop applications at home (so does my father-in-law). OpenJDK not compatible in one instance and not fully compatible in the other.~~
~~I require Sun Java 6 browser plugin for a web applications I use at home. OpenJDK is not compatible.~~
~~A friend of mine requires Sun Java 6 for building AOSP from source. OpenJDK is not compatible.~~
~~Some friends of mine play Minecraft, apparently this will help ;-)~~
~~Janusz Dziemidowicz made it easy for me - https://github.com/rraptorr/sun-java6~~ ~~The PPA currently publishes Sun Java 6 1.6.0.30 for:~~
~~Lucid i386/amd64~~
~~Maverick i386/amd64~~
~~Natty i386/amd64~~
~~Oneiric i386/amd64~~
~~Precise i386. However, amd64 is failing to build on Precise. I will try and fix this in due course.~~

~~To Sun Java 6 , previously installed via packages, do the following.~~

sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get dist-upgrade

~~To install Sun Java 6 JRE do the following:~~

sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-jre

~~To install Sun Java 6 browser plugin do the following:~~

sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-plugin

~~To install Sun Java 6 JDK do the following:~~

sudo apt-add-repository ppa:flexiondotorg/java
sudo apt-get update
sudo apt-get install sun-java6-jdk

~~You can take a look a round my PPA from the URL below:~~

~~https://launchpad.net/~flexiondotorg/+archive/java~~

Installing GNOME 3 on Ubuntu 11.10

martin@wimpress.com (Martin Wimpress) — Fri, 09 Dec 2011 16:21:03 +0000

I tried Unity in Ubuntu Natty 11.04 and Ubuntu Oneiric 11.10. We’ve agreed to hate each other. A few weeks ago I started using GNOME 3 and it only took me a couple of hours to adapt to it’s workflow. GNOME 3 is now my desktop environment at home and and work. I love it! If you’d like to give GNOME 3 a whirl then you could try installing Jan Hoffman’s Ubuntu GNOME Shell Remix from either the 32-bit or 64-bit ISOs he has prepared. This will give a “pure” GNOME 3 experience.

http://ubuntu-gs-remix.sourceforge.net/p/home/

If you already have Ubuntu 11.10 installed then you can install GNOME 3 alongside Unity. Here are the incantations you’ll need to utter in a shell.

sudo apt-add-repository ppa:jan-hoffmann/gnome-shell
sudo apt-add-repository ppa:aegirxx-googlemail/gnome-shell-extensions
sudo apt-add-repository ppa:gnome3-team/gnome3
sudo apt-add-repository ppa:webupd8team/gnome3
sudo apt-get update
sudo apt-get install libglib2.0-bin gnome-core gnome-documents gnome-shell gnome-sushi gnome-tweak-tool gnomeshell-default-settings gtk3-engines-unico

The repositories added above will give you access to Jan’s GNOME 3 meta packages, updated GNOME 3 packages and some extra GNOME 3 extensions. GNOME 3 extensions add all manner of additional tweaks and functionality. Some extensions can even provide a user experience more akin to that of GNOME 2.

http://intgat.tigress.co.uk/rmy/extensions/index.html

In order to get acquainted with GNOME 3 I suggest you read the Discover GNOME 3 (watch the videos too) and GNOME 3 Cheat Sheet pages. Having read those you’ll soon master GNOME 3. After you’ve used GNOME 3 for a while you may conclude it is a more usable desktop environment than Unity, which isn’t a surprising conclusion to arrive at given Unity sucks the big one right now. If you want a “pure” GNOME 3 experience then the following commands will purge Unity and other bits and bobs that GNOME 3 simply doesn’t require.

Remove Unity

sudo apt-purge unity unity-2d unity-2d-launcher unity-asset-pool unity-common \
unity-greeter unity-lens-applications unity-lens-music libunity-misc4

Remove Overlay Scrollbars

These just don’t work on my netbook since they regularly obscure portions of the window I actually want to click on. The can safely be removed even if you intend to continue using Unity.

sudo apt-get purge overlay-scrollbar liboverlay-scrollbar-0.2-0 liboverlay-scrollbar3-0.2-0

Remove Indicators

If you never going back to Unity, Indicators can be safely removed.

sudo apt-get purge xchat-gnome-indicator indicator-appmenu indicator-power \
indicator-session indicator-sound indicator-status-provider-mc5 \
libindicator-messages-status-provider1

Again, Global Menu is not used by GNOME 3. So if you not going back to Unity these can be safely removed.

sudo apt-get purge appmenu-gtk3 appmenu-gtk appmenu-qt firefox-globalmenu \
thunderbird-globalmenu

Finally, a word or warning: Distribution upgrades are not possible! You can’t upgrade to a newer version of Ubuntu when using Jan Hoffman’s Ubuntu GNOME Shell Remix or if you modify an existing Ubuntu 11.10 using my method above. You will have to do a full install once the next Ubuntu release is available. This can’t be fixed as long as Jan’s meta packages are unofficial, because the distribution upgrade process requires having installed one of the desktop meta packages from the official Ubuntu repositories.

Shotwell 0.11 PPA available for Ubuntu Lucid and Maverick

martin@wimpress.com (Martin Wimpress) — Tue, 30 Aug 2011 11:05:50 +0000

Like many others I wanted Shotwell 0.11 for Lucid and Maverick so I’ve created a PPA for it.

https://launchpad.net/~flexiondotorg/+archive/shotwell

My PPA contains Shotwell 0.11 built for Ubuntu Lucid 10.04 LTS and Ubuntu Maverick 10.10. I created this PPA because I run Lucid at home and wanted the new version of Shotwell. Sadly, Yorba aren’t going to provide new Shotwell packages for Lucid due to the reasons discussed in the following ticket: -

http://trac.yorba.org/ticket/3015

As mentioned in the ticket above, there are newer versions of Shotwell available for Lucid in other PPAs. However, those PPAs contain hundreds of packages. If you’re not that brave, like me, then hopefully my PPA provides what you need. I have built Shotwell with minimal changes from the original Yorba source packages and not polluted my PPA with any unnecessary packages. Since Shotwell 0.11 you must enable the GStreamer PPA, see the ticket below for the reasons for this requirement:

http://redmine.yorba.org/issues/3716

To install Shotwell on Lucid and Maverick do the following:

sudo apt-add-repository ppa:flexiondotorg/shotwell
sudo apt-add-repository ppa:gstreamer-developers/ppa
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install shotwell

Enjoy!

Shotwell 0.8.1 PPA available for Ubuntu Lucid

martin@wimpress.com (Martin Wimpress) — Thu, 17 Mar 2011 17:04:42 +0000

Like many others I wanted Shotwell for Lucid so I’ve created a PPA for it.

<https://launchpad.net/~flexiondotorg/+archive/shotwell.

My PPA contains Shotwell 0.8.1 built for Ubuntu Lucid 10.04 LTS. I created the PPA because I run Lucid at home and wanted the new version of Shotwell. Sadly, Yorba aren’t going to provide a Lucid build of Shotwell 0.8.1 due to the reasons discussed in the following ticket:

http://trac.yorba.org/ticket/3015

As mentioned in the ticket above, there are versions of Shotwell 0.8.1 available for Lucid in other PPAs. However, those PPAs contain hundreds of packages. If you’re not that brave, like me, then hopefully my PPA provides what you need. I have built Shotwell 0.8.1 with minimal changes from the original Yorba source packages and not polluted my PPA with any unnecessary packages NOTE! My PPA has dependencies that are satisfied by the Yorba PPA, so you must also enable the Yorba PPA too.

https://launchpad.net/~yorba/+archive/ppa

To install Shotwell 0.8.1 on Lucid do the following:

sudo apt-add-repository ppa:yorba/ppa
sudo apt-add-repository ppa:flexiondotorg/shotwell
sudo apt-get update
sudo apt-get install shotwell

Bash script to retrieve Ubuntu codenames and versions

martin@wimpress.com (Martin Wimpress) — Wed, 16 Mar 2011 13:44:54 +0000

I’m working a script to automatically backport Debian packages to Ubuntu. I needed a way to get a list of currently supported/active Ubuntu releases by codename or version. Here is how I do it.

Get a list of Ubuntu codenames

wget -q http://cdimage.ubuntu.com/releases/ -O - | sed -e :a -e 's/<[^>]*>//g;/</N;//ba' | grep '^[[:space:]][a-z]' | sed 's/\///g'

Get a list of Ubuntu versions

wget -q http://cdimage.ubuntu.com/releases/ -O - | sed -e :a -e 's/<[^>]*>//g;/</N;//ba' | grep '^[[:space:]][1-9]' | sed 's/\///g'

Goodbye F-Spot. Hello Shotwell

martin@wimpress.com (Martin Wimpress) — Sat, 20 Mar 2010 15:39:17 +0000

I’ve finally found a photo organiser for Linux I can live with, Shotwell. Shotwell is a photo organiser for GNOME that I’ve been testing for a few months now but the recent 0.5 release which added tagging and printing, it means Shotwell is finally ready replace F-Spot on my workstation.

Shotwell is intuitive, well documented, extremely easy to use and reliable. It’s easy to dismiss Shotwell as an over simplified photo manager. But once you start using it, you’ll quickly appreciate its clean interface and easy-to-use tools. Don’t just take my word for it either, Shotwell is now the default photo manager in Fedora 13 alpha. Here is a quick run down of the features.

Import photos from folders or from any digital camera supported by gPhoto.
Shotwell automatically groups photos taken at the same time. You can also use tags to organize your photo collection.
You can rotate, crop, reduce red-eye, and adjust the exposure, saturation, tint, and temperature of each photo.
Publish photos to Facebook, Flickr and Picasa Web Albums.

Shotwell provides a non-destructive way to tweak your photos. Instead of modifying the original photos, Shotwell stores all edits in a database and applies them on-the-fly as necessary. This means that you can easily undo all edits. Shotwell comes equipped with all the usual photo enhancing tools and slideshow. You can download a source tarball from the Shotwell home page at http://www.yorba.org/shotwell/ or grab a binary for Ubuntu Karmic or Lucid via Yorba’s Launchpad PPA.

https://launchpad.net/~yorba/+archive/ppa

Give XFS a chance. Don't believe the FUD.

martin@wimpress.com (Martin Wimpress) — Fri, 12 Feb 2010 13:14:44 +0000

After tinkering with Ext4 I did some research and tested other file systems on my new disk arrays. I’ve concluded that XFS, once tuned, is the best file system for my needs and it could well be the best file system for your needs too.

The remainder of this page explains how I arrived at that decision and how I tune XFS to get optimal, yet safe, performance that can rival Ext4 and JFS.

Benchmarks

Here are some benchmarks.

Why I chose XFS

I have not chosen XFS for performance alone, indeed some benchmarks show that XFS it outperformed for some file operations.

My workstation at home has two 6TB disk arrays and a 1TB root file system. The disk arrays contain photo, music and video libraries which are streamed via UPnP/DLNA and DAAP. The video files can be 2GB to 30GB in size. I also do a good deal of HD video encoding, processing and editing. My root partition contains many virtual machine images of which several are running at any given time.

My work laptop has a 250GB root file system and also contains many virtual machine images of which one is usually running.

XFS is designed with large file systems and large file handling in mind. It seems a sensible choice for those reasons alone, but I also liked the following features:

XFS has on-line defragmentation tools, while (at the original time of writing in early 2010) Ext3/4 and JFS do not.
XFS dramatically reduces start-up time by avoiding fsck delay. Ext3/4 can be very slow to fsck large volumes.
XFS has very fast (a few seconds or less) file system creation. JFS is faster than XFS but Ext4 takes many, many minutes.
XFS formatted disk capacity is greater than Ext3/4 even after removing the reserved blocks from the Ext3/4 file system. JFS formatted capacity is similar to XFS.

On that last point, XFS gains 400GB over Ext4 on a 6TB array but when the Ext4 reserved blocks are removed XFS gains 100GB over Ext4.

Tuning XFS

Most of the performance tuning information I found (at the original time of writing in early 2010) is out of date and doesn’t reflect the XFS defaults in modern Linux kernels.

That said, the information on this page is quite old and I no longer feel the need to tweak XFS like I once did.

Creating XFS

XFS 3.1.0 and Kernel 2.6.32 or newer

Ubuntu Lucid 10.04 comes with XFS 3.1.0. The defaults used when creating a XFS file system using Ubuntu 10.04 are optimal and do not require any tweaking.

XFS 3.0.2 and Kernel 2.6.31 or older

Ubuntu Karmic 9.10 comes with XFS 3.0.2. If you are running an earlier Ubuntu release and want to use a tuned XFS root file system you can’t simply use the graphical partitioning tool from the Ubuntu LiveCD installer. However, it is very easy manually create the tuned XFS file systems. Simply boot the Ubuntu Live CD, then start a new shell Application -> Accessories -> Terminal.

Now run the following as root.

mkfs.xfs -l lazy-count=1 -L VolumeName <dev>

lasy-count=1 is a default since XFS 3.1.0 but was recommended by the XFS developers before that. lazy-count is a mkfs option because it changes the on-disk format slightly, and older kernels do not understand this new format. Hence mkfs sets a superblock feature bit to prevent the file system from being mounted on kernels that don’t understand the slightly different disk format. So you must specify lazy-count=0 if you want to disable this feature for older kernels which don’t support it.

http://oss.sgi.com/archives/xfs/2007-12/msg00536.html

Forcing a tuned XFS creation

If you are not sure what XFS version you are running, and therefore what the defaults might be on your system, you can fully tune XFS using the following.

For < 1TB XFS file system

mkfs.xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=4 -L VolumeName <dev>

For > 1TB XFS filesystem

mkfs.xfs -l lazy-count=1,version=2,size=128m -i attr=2 -d agcount=16 -L VolumeName <dev>

Once you have created all your tuned XFS file systems start the Ubuntu installer from the Live CD. When the disk partitioning section comes round choose: Specify Partitions Manually

Now Change each XFS file system telling the partitioner where to mount each XFS file system. But ensure that you do not tick Format the Partition:, thereby preserving your tuned XFS file systems.

When you see this message, just click Continue.

The file system on /dev/sda1 assigned to /boot has not been marked for formatting. Directories containing system files (/etc, /lib, /usr, /var, …) that already exist under any defined mountpoint will be deleted during the install.

Please ensure that you have backed up any critical data before installing.

Mounting XFS

Further performance optimisations can be gained but specifying some additional mount options for your XFS file systems.

To manually mount a XFS file system with, optimal mount options, use the following:

mount -t xfs -o noatime,osyncisosync,logbsize=256k,logbufs=8 <dev> <mtpt>

The /etc/fstab entries I use look something like this.

UUID=xxxxxxxxxxx...x <mtpt> xfs noatime,osyncisosync,logbsize=256k,logbufs=8 0 2

The logsbsize' and logbufsoptions address the often sited limitation of XFS when handling lots of small files and large number of file deletions. The above assumes you don't requireatime. Not using atime` provides a significant performance benefit.

atime, relatime and noatime

Every time a file is accessed (read or write) the default for most file systems is to append the metadata associated with that file with an updated access time. Thus, even read operations incur an overhead associated with a write to the file system. This can lead to a significant degradation in performance in some usage scenarios. Appending noatime to the fstab line for any file system stops this action from happening.

One may also specify a relatime option which updates the atime if the previous atime is older than the mtime or ctime. In terms of performance, this will not be as fast as the noatime mount option, but is useful if using applications that need to know when files were last read (like mutt).

As access time is of little importance in most scenarios, this alteration has been widely touted as a fast and easy way to get a performance boost. Even Linus Torvalds seems to be a proponent of this optimization

http://kerneltrap.org/node/14148

Access time is not the same as the last-modified time. Disabling access time will still enable you to see when files were last modified by a write operation.

async and nobarrier

If you really want to go for all out performance you can also provide async and nobarrier mount options. But you really need to understand and accept the potential issues with using these options.

Read the following to understand what write barriers are and if you are prepared to disable them to gain performance.

http://lwn.net/Articles/283161/

XFS userspace tools

XFS is available as a kernel module in Ubuntu and also available from the Live CDs. Once Ubuntu is installed you can install the XFS userspace tools as follows.

sudo apt-get install xfsdump xfsprogs

De-fragmenting XFS

There are two utilities that XFS has to manage this fragmentation.

xfs_db XFS Debug Information. Used to examine an XFS file system for problems or gather information about the XFS file system.
xfs_fsr File System Organiser. Improves the organisation of mounted file systems. The reorganisation algorithm operates on one file at a time, compacting or otherwise improving the layout of the file extents (contiguous blocks of file data).

Defragment a file system

To find the health of a XFS file system use the xfs_db command to gather some information. In the example below /dev/sda1 is mounted as /boot and /dev/sda3 is mounted as /root.

sudo xfs_db -c frag -r /dev/sda1
actual 162, ideal 162, fragmentation factor 0.00%

sudo xfs_db -c frag -r /dev/sda3
actual 2288833, ideal 254504, fragmentation factor 88.88%

The closer the fragmentation factor is to 0% the better. Unsurprisingly /boot is not fragmented. However /root is very fragmented.

Defragmenting XFS file systems can be done on a live running system, but it is a good idea to schedule this for a time where the partition will be used less.

The file system reorganizer for XFS is xfs_fsr. Typically, I instruct xfs_fsr to reorganise /dev/sda3 with a timeout (-t) of 6hrs (60 * 60 * 6 = 21600) which is specified in seconds. But for the purposes of this example I used a timeout of 15 mins.

sudo xfs_fsr -t 300 /dev/sda3 -v

The output will look something like this.

/ start inode=0
ino=145565
extents before:2 after:1 DONE ino=145565
ino=145662
extents before:2 after:1 DONE ino=145662
ino=600148
extents before:2 after:1 DONE ino=600148
ino=1127295
extents before:82794 after:1 DONE ino=1127295
ino=1127243
extents before:2 after:1 DONE ino=1127243
ino=1382852
extents before:50869 after:1 DONE ino=1382852
ino=1422636

When the defrag is finished check how well the file system reorganising was.

sudo xfs_db -c frag -r /dev/sda3
actual 2155648, ideal 254512, fragmentation factor 88.19%

As you can see defragmenting for 15 mins doesn’t improve things greatly, which is why xfs_fsr needs to be run for several hours or more.

Manually defragmenting the file system is simple enough, but a better solution would be to schedule a cron job to run periodically.

Defragment a file

It is also possible to de-fragment a single file. To determine if a file is in need of defragmenting run the following…

xfs_bmap -v /srv/A320/PGQAR.DAT | wc -l

This will output a number which showing the number of extents the file is using.

This number should be close to 1. So in the example above, I have a very fragmented file.

sudo xfs_fsr -v /srv/A320/PGQAR.DAT

This will output something like the following.

/srv/A320/PGQAR.DAT
extents before:95278 after:1 DONE /srv/A320/PGQAR.DAT

The file is now defragmented. I use the method above to target defragmentation where I know files reside that are most likely to be fragmented, rather than defragmenting the whole file system.

References

XFS References

Performance Tuning XFS References

Recovering reserved space from ext4

martin@wimpress.com (Martin Wimpress) — Thu, 07 Jan 2010 20:42:04 +0000

The Ext4 file system, like Ext3, reserves 5% of the blocks on the file system for the root user. The reserved blocks are there for root’s use as a safe guard if the filesystem gets full, it provides some wiggle room to enable the really important programs to still function. But in some cases there’s not much point in having space reserved for root. I’ve recently upgrade my workstation with a 6TB internal RAID 0 array for data storage (music, videos, photos, etc) and an external 6TB RAID 0 array as a backup. My OS boot from a 1TB drive. For my 6TB arrays I want the maximum available storage and was interested to see what effect removing the reserved space would have. So, this is what I did. First I made the Ext4 file system, mounted it and queried how much space was available.

sudo mkfs.ext4 /dev/sdh1
sudo mount /dev/sdh1 /mnt
df -h

Looks like I have 5.1TB of available space.

/dev/sdh1             5.4T  186M  5.1T   1% /mnt

Then I unmounted the file system, removed the reserved blocks, checked the consistency of the file system, mounted it and queried how much space was available.

sudo umount /mnt
sudo tune2fs -m 0 /dev/sdh1
sudo e2fsck /dev/sdh1
df -h

Looks like I have 5.4TB available now, a saving of 300GB.

/dev/sdh1             5.4T  186M  5.4T   1% /mnt

Now, I could have simply created the files system without the reserved blocks in the first place, but I was interested to see the comparison.

sudo mkfs.ext4 -m 0 /dev/sdh1

Before you start removing the reserved blocks from your ext3/ext4 file systems do a bit a research first.

StarTech S354UER Review

martin@wimpress.com (Martin Wimpress) — Sat, 02 Jan 2010 16:00:25 +0000

*** UPDATE: The StarTech S354UER completely died after less than one year. Not recommended! ***

I’ve ripped my entire CD collection to MP3 and I’m in the process of ripping my entire DVD, Blu-Ray and HD-DVD collection to MPEG-2 TS files so that I can stream everything to my PS3 using MiniDLNA. I currently have this data stored on an internal 2TB volume and backed up to an external 2TB volume. I currently have just 360GB remaining capacity and I’ve only imported half my DVD collection and one Blu-Ray. I need more storage.

I wanted to keep the same backup method, large internal volume backed up to a large external volume of the same size. I opted for Samsung Spinpoint F2 EcoGreen (HD154UI) drives because they are relatively inexpensive, low power (therefore lower heat) and quiet.

I decided to get 4x 1.5TB drives for the internal volume and stripe them to give 6TB of storage and I went looking for an external box in which I could install 4x 1.5TB drives and also stripe or span them. That meant a multi disk external enclosure, with some kind of RAID, supported by Linux and that isn’t too expensive. A tall order as it turns out.

After lots of research I finally found the StarTech S354UER which on paper appears to do what I required and a good deal more.

Compatible with Windows, Mac, and Linux operating systems
Fan control button to enable manual control of the fan and switch between the three fan speeds
Internal three speed 80mm fan with automatic or manual controls
Multiple LED indicators to provide RAID information, hard drive activity, HDD Status, RAID rebuild status, fan settings, and interface in use
No software required
Package includes 1x USB, 1x eSATA, 1x FireWire 400, 1x FireWrie 800 cable, Power adapter and cord, and the manual
Plug-and-Play and Hot swap supported with USB 2.0, eSATA, and FireWire
Push button raid configuration eliminated the need to disassemble the enclosure to upgrade your raid configurations
Removable front cover for easy access to hard drive
Rugged aluminum chassis
Supported File Systems:NTFS, FAT, FAT32, and ext3
Supports four 3.5in hard drives up-to 2.0 TB each in size
Supports RAID 0, RAID 1, RAID 3, RAID 5, RAID 10 (RAID 1+0), and Spanning

Normally, I will read reviews of different products and select something with a proven track record particularly when Linux support is required. I couldn’t find much in the way of reviews for the StarTech S354UER so I took a gamble and decided to buy one. Eeek!

In short it works and it is quiet. It is currently sitting no more than 50cm from me initialising a stripped array of 4x1.5Tb disks as Ext4. I can’t hear it but I have manually set the fan speed to low using the fan control buttons on the front on the chassis.

The build quality is not great, but not awful either, but once the drives are installed and clamped in place they are very secure. It is impossible to tell if you’ve pushed the power button you have to wait and see if the device powers up/down to be sure. The fan speed controls work, but are inverted from what is documented in manual. Fan1 is documented as LOW in the manual but is actually HIGH.

Setting up the device was not quiet plug and play either but the issues I ran into may not be entirely the fault of the StarTech S354UER. My plan was to connect the enclosure via Firewire and as yet I’ve not been able to get the enclosure to be recognised via Firewire using Ubuntu Jaunty 9.04. However, I am a Firewire newbie so maybe more research required. I don’t have eSATA (yet) so I have the device connected via USB 2.0. Which does work.

One of my new hard disks turned out to be DOA. It took me a while to figure out what was wrong here. The StarTech drive failure light on the front of the chassis was illuminated, but I didn’t know how to tell which drive had actually failed. After some trial and error I found that there are four internal LEDs, one for each disk. Starting the StarTech with the chassis door open you can see the internal LEDs blinking as each disk is spun up and tested. If the drive failure LED on the front of chassis is illuminated look at the internal LEDs, the drive LED which is off denotes the failed drive. This information is not in the user manual!

I replaced the drive and was able to select my RAID level. Selecting the RAID level is done though a combination of DIP switches under a panel at the back of the unit and buttons on the front. It is a slightly fiddly process, but it does have the advantage that you can’t accidentally change your RAID levels and re-initialise the array.

In order to create a partition greater than 2TB you have to use GPT. I’d not encountered GPT before, but I found everything I needed to know on the page below.

Make the most of large drives with GPT and Linux

I used gparted to create my partition and format with Ext4 with the enclosure connected via USB 2.0. The whole process took ~1 hour. As you will see from the link the quoted price makes this enclosure pretty expensive. Search around though, because I got mine of 50% less than the price quoted on the StarTech.com website. You do get a healthy selection of RAID levels, all the cables, screws, screwdriver and drive handles you require.

Would I recommend the StarTech S354UER? Time will tell, but it does work with Linux via USB 2.0 and I will continue investigate FireWire and I may add eSATA in the future to see what the performance benefits are. But for what I bought it for, secondary storage for backups, it is a pretty cheap way to add a multi terrabyte array to your system.

Mediatomb vs. MiniDLNA

martin@wimpress.com (Martin Wimpress) — Fri, 18 Dec 2009 15:34:12 +0000

I’ve been using Mediatomb for nearly two years now but decided to give MiniDLNA a whirl since it is a fully fledged DLNA server whereas Mediatomb is UPnP only. I’m currently running both Mediatomb SVN and MiniDLNA CVS. So, how does MiniDLNA compare to Mediatomb?

MiniDLNA is easier to compile, configure, uses less RAM and has less software dependencies than Mediatomb.
MiniDLNA doesn’t currently support music play lists or Last.fm scrobbling. Mediatomb supports .m3u and .pls playlists but requires a 3rd party patch to add Last.fm scrobbling.
MiniDLNA doesn’t support dynamic video thumbnail creation, which would be nice to have but is not essential, cover images are supported. Mediatomb supports video thumbnails via ffmpegthumbnailer.
MiniDLNA doesn’t currently have any transcoding support. This is of little consequence for me since I import video content into my library in a format natively supported by the PS3, either MP3, MPEG-2 TS or MPEG-4. Mediatomb does support transcoding but it is somewhat fiddly to setup and you can’t pause transcoded content.
MiniDLNA works “out of the box” with the PS3 (and other DLNA clients) while Mediatomb requires some tweaking.
Mediatomb’s default video import script doesn’t suit how I organise my video library, but MiniDLNA suits my video library perfectly.

So, as of today I am running both Mediatomb and MiniDLNA. Mediatomb is exclusively handling audio since playlist and Last.fm support are essential for me. MiniDLNA is now handling video exclusively. I’m very happy with the results but should MiniDLNA add .m3u/.pls play lists and Last.fm support I will switch everything to MiniDLNA.

2 years later…

Since writing this post MiniDLNA added support for playlists. It still doesn’t support Last.fm scrobbling though. Despite that I switched to MiniDLNA and it has been streaming audio and video around the house for that last couple of years.

Amazon Loves Linux Music Lovers

martin@wimpress.com (Martin Wimpress) — Wed, 16 Dec 2009 22:30:48 +0000

I’ve no idea when Amazon.co.uk launched their MP3 store and I’ve no idea when they released their Linux client for downloading the MP3s you purchased. I don’t care, I just want to say I’m really impressed Amazon have considered us Linux users. Well done Amazon!

Not only that but the MP3s are DRM free, encoded using variable bit rates aiming at an average of 256 kilobits per second (kbps), album cover art is included with each song and the tracks are typically cheaper than iTunes. Well done again.

Doubtless some would want an Open Source client and unencumbered formats such as Ogg and FLAC, but I’m pretty happy with what Amazon have on offer so long as it works. Which it does.

However, the Linux MP3 downloader client is 32-bit only. Not so good, but it can be successfully installed in 64-bit Ubuntu. Here’s how I did in on Ubuntu Jaunty 9.04 64-bit.

wget -c http://frozenfox.freehostia.com/cappy/getlibs-all.deb
dpkg -i getlibs-all.deb    
wget "http://www.amazon.co.uk/gp/dmusic/help/amd-installer-redirect.html/ref=dm_amd_linux_ubuntu?ie=UTF8&forceos=LINUX&callingPage=%2Fgp%2Fdmusic%2Fhelp%2Famd.html&linux_Ubuntu.x=1" -O amazonmp3.deb
dpkg -i --force-architecture amazonmp3.deb
getlibs /usr/bin/amazonmp3

DVD to MPEG2-TS Ripper for Linux

martin@wimpress.com (Martin Wimpress) — Fri, 04 Dec 2009 07:53:58 +0000

A while back I released a script that rips a DVD to MPEG-2 PS allowing the user to select one audio stream and one subtitle stream. Optionally the video can be requantised, using M2VRequantiser and an ISO image created. If creating an ISO image the chapters are also preserved from the original DVD. You can see the original post below.

DVD to MPEG-2 PS Ripper for Linux

I’ve just released an update to that original script which fixes subtitles in the original MPEG-2 PS mode but now adds the capability to rip MPEG-2 TS. The video stream can still be shrunk and in MPEG-2 PS mode the video is still requantised but in MPEG-2 TS mode the video is re-encoded as H.264.

Requantising is faster but can introduce artifacting. H.264 encoding is slower, but produces very good quality. I am currently re-importing my entire DVD collection, using this script, to my DLNA server using MPEG-2 TS and re-encoding the video to H.264. This gives me high quality rips at relatively small size (~3Gb) whilst preserving Dolby Digital 5.1 audio. Perfect for playback via DLNA on the PS3. Some things to be aware of:

Subtitles are only supported in MPEG-2 PS mode.
MPEG-2 PS files created by this script are DVD compliant.
ISO files created by this script will preserve the chapters from the original DVD.
The PS3 can only play DTS audio in MPEG-2 PS streams when they have been authored to DVD.
The PS3 can only play subtitles in MPEG-2 PS streams when they have been authored to DVD.
The PS3 can’t play DTS audio in MPEG-2 TS streams therefore this script will transcode DTS to AC3 when in MPEG-2 TS mode.

To download the script and find out how to make full use of it visit the release page below.

DVD-to-MPG

Linux Hex Editors

martin@wimpress.com (Martin Wimpress) — Thu, 03 Dec 2009 11:59:30 +0000

We use hex editors daily at work, we are regularly cutting up data from flight data recorders for analysis or recovery. So when I find a new hex editor for Linux I usually give it a try. I happened across a blog listing five GUI hex editors for Ubuntu today. Find out more below.

Five gui hex editors for ubuntu

Creating a JetDirect Server with Linux

martin@wimpress.com (Martin Wimpress) — Sat, 17 Oct 2009 11:41:21 +0000

I created JetDirect compatible server on my NSLU2 running Ubuntu Jaunty 9.04 using p910nd, which is a small printer daemon that does not spool to disk but passes the job directly to the printer. It is particularly useful for disk less Linux workstations and embedded devices that have a printer hanging off them.

PS3, Mediatomb, Multi Zone Music streaming

martin@wimpress.com (Martin Wimpress) — Sat, 26 Sep 2009 11:02:52 +0000

PlayStation 3 firmware 3.00 added a new feature I was very excited about, multi-av output. Today I finally got round to re-wiring the home cinema system to make use of this new feature. I now have the PS3 streaming music from MediaTomb with my A/V receiver sending audio to Zone 1 via digital inter connects and also sending audio to Zone 2 via analog stereo interconnects.

Zone 1 is a 5.1 speaker setup and Zone 2 is a 2.0 all weather wireless speaker system which are often in the kitchen but also moved outside for parties. If Sony could just add Skype to the PS3 and allow the PSP Remote Play to output audio to both the PS3 and PSP (rather than just one of them) I would be very happy indeed.

IMDB Film Summary as a MPEG-2 video, Part 2

martin@wimpress.com (Martin Wimpress) — Sun, 20 Sep 2009 08:48:36 +0000

UPDATE! I no longer use or maintain the script below. I suggest the vastly superior Sheet Maker for Linux.

Back in April I released a script to create a MPEG video summarising a movie using data from IMDB, you can find the original post in the URL below to learn why I created such a script in the first place.

IMDB Film Summary as a MPEG-2 video

Today I’ve released v2.0 of that script, which is almost a complete re-write mostly thanks to Eric at yPass.net who contributed significantly. Thanks to Eric the script is much improved since version 1.0, here is a run down of what’s new.

v2.0 2009, 19th September.

Merged yet more contributions from Eric, http://www.ypass.net. Thanks Eric!
Added usage instructions.
Added categorisation by Certificate.
Added dynamic computation of video bitrate.
Added silent audio generation.
Added a shell script to reprocess an entire film store.
Re-added MPEG-2 video encoding.
Improved video encoding speed by removing pre-processing with jpeg2yuv.
Fixed spiffy animations when cover art is not available.
Fixed spiffy animations on platforms that may have incomplete GD.
Modified filename input so that an input filename is optional rather than mandatory.

v1.2 2009, 17th July.

Merged extensive contributions from Eric, http://www.ypass.net. Thanks Eric!
Updated the README to reflect Eric’s changes.
MPEG-4 video encoding replaced MPEG-2 video encoding.
Never released to the public.

To download the script and find out how to make full use of it visit the release page below.

IMDB-to-MPEG

As it stands the IMDB-to-MPEG script does what I require, so I will maintain it in it’s current form. However, Eric has been working on a new direction by adding support for NetFlix, creating a GUI with php-gtk and some other cool stuff. While Eric has shared the details with me, I simply don’t have the time to add all that good stuff to IMDB-to-MPEG, so if you like the sound of what Eric has been up to hop over to his site to find out more.

Netflix Has a Developer API

MKV to MPEG-4 conversion script

martin@wimpress.com (Martin Wimpress) — Thu, 27 Aug 2009 20:48:58 +0000

The PlayStation 3 can’t play MKV files. Therefore I’ve written a script that creates a PlayStation 3 or Xbox 360 compatible MPEG-4 from Matroska providing the video is H.264 and audio is AC3 or DTS.

Xbox 360 compatibility requires that audio is forcibly downmixed to stereo with --stereo. AAC 5.1 audio will have the correct channel assignments when transcoding from AC3 5.1 and DTS 5.1. If neroAacEnc is installed then it is used in preference to faac for encoding the AAC audio, as it produces better quality output. neroAacEnc is optional.

The script does as little re-encoding as possible, only the audio and subtitles are re-encoded or converted. The script can detect profile 5.1 H.264 and patch it to 4.1 in under a second. Any subtitles in the Matroska are preserved. If mp4creator is used the subtitles are extracted stored in a seperate file. If MP4Box is used (default) the subtitles are converted to GPAC Timed Text and muxed into the resulting MPEG-4. The PlayStation 3 can’t display these subtitles but some software players can.

The script can optionally split the Matroska if it is greater than 4GB to ensure PlayStation 3, Xbox 360 and FAT32 compatibility. This script works on Ubuntu and should work on any other Linux/Unix flavour and possibly Mac OS X providing you have the required tools installed.

MKV-to-MP4

MP3Gainer - Apply ReplayGain to your entire music library

martin@wimpress.com (Martin Wimpress) — Fri, 14 Aug 2009 15:25:45 +0000

Work has been crazy. We’re moving house. Hence, not much time for geeky stuff recently. I’ve been putting this off for ages, I need to “normalise” the volume of my MP3 music music library. Not all CDs sound equally loud. Whilst different musical moods require that some tracks should sound louder than others, the loudness of a given CD has more to do with the year of issue or the whim of the producer than the intended emotional effect. This difference carries over when you rip the CD to MP3 and random play through my music collection requires constant manual volume adjustment. This has been bugging me for a while now, but when it started to bug my wife I knew it was time to find a solution. My main concerns with applying some sort of audio normalisation were….

My MP3s should not be irretrievably changed into something I end up hating.
The method used should be free of the application used for music playback, given that I play my music on iPod Nano, iPod Shuffle, PSP, PS3, Linux desktops, TomTom 720T FM streaming and in car MP3 player.

After some research mp3gain seems to be the tool for the job which provides an implementation of ReplayGain. However, as of today my entire CD collection is ripped, which is very large, so I needed a way to process my whole music collection in an automated fashion. I found some examples of how to script this, but there are caveats with the solutions I found. Therefore I have created my own script, MP3Gainer, to apply ReplayGain using mp3gain which overcomes these common limitations. MP3Gainer recursively applies ReplayGain to a MP3 music collection of any size and directory depth. ReplayGain can be applied in ’track’ or ‘album’ mode and if ReplayGain has previously been applied it can also be undone. It is important to understand that MP3Gainer ‘album’ mode really is per album, which is what you want. Trust me! This script works on Ubuntu, should work on any other Linux/Unix flavour and possibly Mac OS X providing you have the required tools installed.

MP3Gainer

PlayStation 3 compatible MPEG-4 container repacker

martin@wimpress.com (Martin Wimpress) — Thu, 23 Apr 2009 17:09:11 +0000

Some of my mobile phones have been able to record video clips in MPEG-4 format. Sadly some of these clips don’t play on the PlayStation 3 and those that do stutter terribly. I use iplayer-dl to download content from BBC iPlayer. Sadly the files are in a Quicktime container and are not playable on the PlayStation 3.

In order to address both these issues I created a script which extracts the audio and video from an existing MPEG-4 or ISO Media Apple QuickTime container and repacks them in a new MPEG-4 container with optional splitting of the resulting MPEG-4 to maintain FAT32 compatibility. The new MPEG-4 files play just fine on my PlayStation 3. This script works on Ubuntu, should work on any other Linux/Unix flavour and possibly Mac OS X providing you have the required tools installed.

MP4-Repacker

MKV to M2TS conversion script

martin@wimpress.com (Martin Wimpress) — Thu, 23 Apr 2009 17:00:58 +0000

The PlayStation 3 can’t play MKV files. Therefore I’ve written a script that creates a PlayStation 3 compatible M2TS from a MKV, assuming video is H.264 and audio is AC3 or DTS with as little re-encoding as possible. Any subtitles in the MKV are preserved in the M2TS although the PlayStation 3 can’t display subtitles in M2TS containers. Optionally splits the M2TS, if it is greater than 4GB, to maintain FAT32 compatibility. Unlike other MKV to M2TS solutions, this script doesn’t create any intermediate files during the conversion.

The PlayStation 3 can’t play DTS audio streams in M2TS containers, therefore DTS audio is transcoded to AC3.

This script works on Ubuntu, should work on any other Linux/Unix flavour and possibly Mac OS X providing you have the required tools installed.

MKV-to-M2TS

DVD to MPEG2-PS Ripper for Linux

martin@wimpress.com (Martin Wimpress) — Thu, 23 Apr 2009 16:50:31 +0000

Every so often I find myself in looking through the ex-rental DVD “bargain bin”. Quite often I find something I consider a bargain. However, the experience of watching an ex-rental DVD is typically ruined by the various trailers and marketing guff at the start which you can’t skip. My wife hates that stuff, and I love my wife, so I routinely rip the main feature of newly acquired ex-rental DVD movies so we can avoid that crap.

I run a Mediatomb DLNA server and I want to load it with all my DVDs. Ripping them helps reduce the amount of storage I require. MPEG2-PS files are compatible with my PlayStation 3 which is the client to my Mediatomb DLNA server. As a solution to the above I created a script, which can extract the main feature from a DVD video, allowing the user to select one audio stream and one subtitle stream. Optionally the video can be requantised, using M2VRequantiser, and an ISO image created. If creating an ISO image the chapters are also preserved from the original DVD. I’ve lobbed my code into GitHub.

DVD-to-MPG

M2VRequantiser for 32-bit and 64-bit Linux

martin@wimpress.com (Martin Wimpress) — Thu, 23 Apr 2009 16:47:20 +0000

I recently discovered that tcrequant (part of the transcode suite of tools) has been deprecated. Worst still I found that when I ran tcrequant on my 64-bit Linux workstation is was corrupting the video. See the links below for details.

Therefore I decided to get the M2VRequantiser code from Metakine working on both 32-bit and 64-bit Linux as a replacement for tcrequant. M2VRequantiser accepts the raw MPEG2 video data (not VOB) from the standard input and writes the recompressed frames to the standard output. M2VRequantiser takes two arguments. The first one is a floating point value specifying the ratio of compression. The second is the size of the M2V, since the data is streamed to M2VRequantiser it cannot know the M2V size. The following command would recompress ‘original.m2v’, whose size is 1024000 bytes, by a factor of 1.25.

M2VRequantiser 1.25 1024000 < original.m2v > requantised.m2v

I’ve only tested on 32-bit and 64-bit Linux, specifically Ubuntu 8.10. It works for me but I’d be interested to get your feedback.

M2VRequantiser

IMDB Film Summary as a MPEG-2 video

martin@wimpress.com (Martin Wimpress) — Wed, 22 Apr 2009 18:00:07 +0000

UPDATE! I no longer use or maintain the script below. I suggest the vastly superior Sheet Maker for Linux.

If you’ve read my blog before you’ll know I run Mediatomb DLNA server with my PlayStation 3 as the client, You’ll also know I am working towards importing my entire DVD collection into my Mediatomb server. However, my wife wants to know something about each film in the library without having to dig out the DVD case from storage. My solution is to include a MPEG-2 video displaying the film summary in the Mediatomb library for each DVD I have imported so it can be easily viewed from the PS3.

My script is called IMDB-to-MPEG and I’ve finally got round to uploading it.

IMDB-to-MPEG

The scripts takes one parameter as input, a film title. The plotline, year of release, genres, cast list and running time for that film are gathered from IMDB and formatted as text. Here is an example.

 The Usual Suspects (1995)

 A boat has been destroyed, criminals are dead, and
 the key to this mystery lies with the only
 survivor and his twisted, convoluted story
 beginning with five career crooks in a seemingly
 random police lineup. (106 mins)

 Starring Stephen Baldwin as Michael McManus,
 Gabriel Byrne as Dean Keaton, Benicio Del Toro as
 Fred Fenster, Kevin Pollak as Todd Hockney, and
 Kevin Spacey as Roger 'Verbal' Kint.

 Genres: Crime, Mystery, Thriller.

 Rated 8.7 out of 10 from 227,964 votes.

The text is converted into an image and then encoded into a MPEG-2 video using the lowest possible bitrate/resolution that is acceptable to read when viewing on a 42" plasma from my sofa.

Directories for each matching genre are created and also one for the IMDB rating (rounded down). The MPEG-2 is stored in the ‘All’ folder and then symlinked to the genres and rating for that film. I then copy my video into the appropriate directory in ‘All’. For example.

 .
 |-- All
 |   `-- The_Usual_Suspects
 |       `-- About_The_Usual_Suspects.mpg
 |-- Genres
 |   |-- Crime
 |   |   `-- The_Usual_Suspects -> ../../All/The_Usual_Suspects
 |   |-- Mystery
 |   |   `-- The_Usual_Suspects -> ../../All/The_Usual_Suspects
 |   `-- Thriller
 |       `-- The_Usual_Suspects -> ../../All/The_Usual_Suspects
 |-- Ratings
 `-- 8
 `-- The_Usual_Suspects -> ../../All/The_Usual_Suspects

This code was lashed up in a few hours, it ain’t pretty but it works for me on my Ubuntu systems, maybe it’ll work for you too.

Converting Matroska to M2TS for PS3 and Mediatomb

martin@wimpress.com (Martin Wimpress) — Wed, 15 Apr 2009 16:19:24 +0000

It has been a while since I last posted, mainly due to not having Internet access at home for a month. Anyway, I’m online again and I have been tinkering with various projects the most recent of which is Matroska conversion (again).

Matroska to MP4

For sometime I have been converting Martoska files to MPEG-4 with AAC 5.1 audio so I can stream them via Mediatomb to my PlayStation 3. The conversion process works well although there is some overhead in transcoding the audio and the AAC 5.1 audio is not as good quality as the original AC3 or DTS.

If you are interested I’ve put my code in GitHub, the script automates the whole process.

MPV-to-MP4

Matroska to M2TS

A little while back I read it was possible to convert those same Matroska file to M2TS files which, so long as the audio is AC3, so takes much less time to convert. As the PlayStation 3 can’t play DTS audio streams inside a M2TS container there is still a requirement to transcode DTS to AC3. That said the conversion to M2TS requires less file I/O than converting to MPEG-4 and is therefore it is generally a quicker conversion method, typically just 2 or 3 minutes on my workstation at home.

Plus the audio quality of the AC3 or transcoded DTS is better than that of transcoded AAC 5.1. I’ve created my own script to fully automate the conversion process. The script has been tested on Ubuntu 8.10 64-bit but there is an outside chance it will work on Mac OS X if you can get the required tools installed. Again, you can find my script on GitHub.

MPV-to-M2TS

Mediatomb 0.12 - Streaming audio and video around the house

martin@wimpress.com (Martin Wimpress) — Sat, 07 Feb 2009 16:23:39 +0000

Mediatomb is an open source (GPL) UPnP MediaServer with a nice web user interface, it allows you to stream your digital media through your home network and listen to/watch it on a variety of UPnP compatible devices, such as the PlayStation in my case.

Mediatomb 0.12 is not yet released as final yet but it is certainly stable enough for general use, so I spent the last week migrating from Mediatomb 0.11 to Mediatomb 0.12.

I’ve recently finished ripping my entire CD collection (344 CDs) to MP3 and I am currently ripping my DVD collection (85 done so far) to MP4 with AAC 5.1 audio. The ‘Music’ and ‘Video’ folders in our home directories are mounted via NFS. The Mediatomb server uses the same data sources so any playlists or new music/videos we might import are immediately reflected in Mediatomb.

Our entire CD library is now available at the click of a button, automatically organised by genre, artist and date. We have also created some playlists in .m3u or .pls format.

New to Mediatomb 0.12 is the ability to scrobble your music to Last.fm, this a killer feature for me and why I chose to migrate to 0.12 before it goes final.

I ripped the CDs using SoundJuicer, since I can configure it to use LAME presets. I then used the Music Brainz Picard Tagger to add additional tagging and embed cover art and then applied ReplayGain.

Finally my wife and I use Banshee to manage the music library on our computers, including the creation of playlists and syncing to our iPods.

I am using Handbrake to rip the DVDs to MP4. I’ve created a new PS3 compatible profile which is focused on quality, I’ll post details about that in the future. Mediatomb 0.12 has some experimental features to stream video content from .ISO images of DVDs. I’ve yet to play with that but it sounds very cool. I’ve also created a script which queries IMDB to categorise our film library by genre and create summary information about each film in the library. I’ll be posting more about that soon. I haven’t finalised how we will integrate Photo management with Mediatomb yet, but that is that final piece in the puzzle.

Distributing closed source Python applications

martin@wimpress.com (Martin Wimpress) — Thu, 29 Jan 2009 13:04:03 +0000

Last November we switched to Python as the principal language for all new software development projects at work, ditching Microsoft Visual C++ and PHP in the process. Last Friday we released our first Python application to our customers for both Windows and Linux users.

Although we make good use of Open Source software development tools and methodologies the application we have just written is propritary and the source can not be released publicly. We needed to compile, or freeze, the Python script in order to create a standalone executable. Tools that do this have been around for sometime, however our application makes use of Win32 Extensions for Python and WMI on Windows, DBUS/HAL on Linux, wxPython on both, and a number of other modules. This is quite a big ask for the Python script compilers and initially the only tool which could build this lot successfully was py2exe. Sadly that only solves part of the problem since it is a Windows only tool.

Then we found bbfreeze, which supports both Windows and Linux with Mac OS X support being actively developed. bbfreeze has a simple build API and we were soon using it to build stand alone executables for both Windows and Linux. Everything is peachy, all we needed was as means to install our application.

We only need a tarball for Linux since we manage all kiosk installations, but our customer can install the Windows version. Enter InnoSetup. InnoSetup is a free installer for Windows programs and installer can even be created from the command line, perfect for integration with our Jenkins build servers.

Wordpress Search Engine Optimisation

martin@wimpress.com (Martin Wimpress) — Mon, 26 Jan 2009 15:37:55 +0000

I manage several websites at work and have therefore learned something about search engine optimisation (SEO). Our main site at work uses Wordpress as the content manager, which has good SEO features by default. Last week we launched a new website and I also refreshed the other sites, in so doing I found a great article discussing how to fully SEO your Wordpress site. It is a good read even if you don’t use Wordpress, but for those of you that do, it recommends a suite of plugins that do all the hard work for you.

Wordpress SEO - The definitive guide to high rankings for your Blog

Hacking WordTwit into WordDent

martin@wimpress.com (Martin Wimpress) — Fri, 23 Jan 2009 18:24:20 +0000

I have quickly forked WordTwit into WordDent. WordDent is a Wordpress plugin that utilizes the Twitter API to automatically push a published post to your Identi.ca account as a dent. It allows all your Identi.ca contacts to keep up to date with your blog postings. I’m still just testing that WordDent works correctly. I’ll find out if the nice people over at Brave New Code will agree to me releasing WordDent since I can’t fine any license details for WordTwit.

Converting DVB-T to DVD Compliant MPEG-2

martin@wimpress.com (Martin Wimpress) — Thu, 22 Jan 2009 23:46:26 +0000

I am just about to clean up and convert another batch of programmes I have recorded from Freeview (DVB-T in the UK) so that I can add them to my DLNA Server. I thought I’d share the method I use on Ubuntu.

By clean up, I mean edit out any adverts and trim crap from the start and the end of the recordings. It just so happens that the result of this process is a DVD compliant MPEG-2 which is suitable for DVD authoring, or in my case, streaming around the house. This method of conversion should work for any DVB PVR which allows you to export recordings via USB and, of course, MythTV or similar.

DVB Ripping

I have a PlayTV add-on for the PlayStation 3 which enables me to record Freeview (DVB-T) broadcasts to the PS3 internal hard disk. I mostly use PlayTV to record films. To prevent the PlayStation 3 hard disk filling up with films I wanted to export, edit out any adverts and then serve the edited file from my DLNA server or author it to DVD.

This process does not re-encode the audio or video therefore it is fairly quick and the output is the same quality as the input.

Although I am using a PlayStation 3 as PVR and MPEG-2 TS (Transport Stream) file can be converted to a MPEG-2 PS (Program Stream) file using this process.

Export from PlayStation 3

First we need to move the recording from the PlayTV Library to the PS3 Home Menu. Start PlayTV, open the Library, select the recording and choose the Move to Home Menu option.

Quit PlayTV
Plug in an external (FAT32 formatted) USB drive to the PS3.
Go to Video on the PS3 Home Menu and select the recording you moved there earlier.
Select Copy from the Options screen and choose the external USB drive as the target.

Clean MPEG-2 TS and convert to MPEG-2 PS

Plug the USB drive into your Ubuntu workstation and copy the .m2ts file to your hard disk.

You will need to Project X to clean the MPEG-2 TS and convert it to MPEG-2 PS.

apt-get install project-x

Edit out the adverts

Start the Project X GUI and load your .m2ts file.

File -> Add and select your .m2ts file.

Now use Project X to add cut points to edit out any adverts.

De-multiplex the audio and video

When you have completed your edits you need to ‘demux’ the .m2ts file into two streams, one holding the audio (.mp2) and one holding the video (.m2v).

Click the Prepare >> button.
From the Process Window select the Action type to M2P
Click the start button and wait for the processing to finish.
Clock the ‘‘Process Windows’’ and quit Project X.

Re-multiplex the audio and video

The reason for the de-mux and then re-musing it to ensure the timecodes are correct, other the video will not playback correctly.

Install MJPEG tools.

apt-get install mjpegtools

Now we need to re-multiplex the audio and video to create a DVD compliant MPEG-2 PS file.

mplex -f 8 -o muxed-%d.mpg audio.mp2 video.m2v

The -f 8 option specifies a dvd-compliant stream that is compatible with dvdauthor. The -o option specifies the outfile, you can substitute muxed-%d.mpg with a more descriptive name if you like. %d is expanded to a number if mplex decides to split the output to several files, this usually happens when the recording contains commercials and is nothing to worry about.

Author DVD

The MPEG-2 PS file that has been created should be suitable for DVD authoring using DeVeDe. When adding MPEG-2 PS files created using the method above open the DeVeDe Advanced options and select This file is already a DVD/xCD-suitable MPEG-PS file in the Misc menu.

References

Sync Zimbra Contacts, Calendar and Tasks to your iPod

martin@wimpress.com (Martin Wimpress) — Wed, 21 Jan 2009 13:41:51 +0000

We use Zimbra at work for email, contacts, calendaring, etc. I have Zimbra syncing with Thunderbird and my phone, I love it.

At the weekend I was updating the music library on my iPod Nano (2nd Gen) and noticed I had hidden the Contact and Calendar menu entries. I decided to see if I could get my iPod Contacts and Calendar synced with Zimbra, it turned out to be very simple. In the examples below replace username and password with your Zimbra user credentials. Obviously use the URL to your Zimbra server and replace /media/IPOD with where your Linux distribution has mounted your iPod.

wget https://username:password@your.zimbraserver.tld/zimbra/home/username/contacts.vcf -O /media/IPOD/Contacts/contacts.vcf
wget https://username:password@your.zimbraserver.tld/home/username/Calendar -O /media/IPOD/Calendars/calendar.ics
wget https://username:password@your.zimbraserver.tld/home/username/Tasks -O /media/IPOD/Calendars/tasks.ics

VMWare to VirtualBox Migration

martin@wimpress.com (Martin Wimpress) — Mon, 19 Jan 2009 17:52:49 +0000

Last week I switched from VMware to VirtualBox at home. Why? Well, hack value mostly and I also wanted to learn more about VirtualBox having never used it before.

Bottom line, for home use it suits me very nicely and is far easier to get installed and running when compared to VMware Server which almost always required patching. The VirtualBox and Guest Additions (think VMWare Tools) installers both worked without a hitch. I even migrated my existing VMware Linux guests to VirtualBox, again everything went smoothly and my existing .vmdk disks worked just fine.

Search engine verification and sitemaps

martin@wimpress.com (Martin Wimpress) — Sun, 18 Jan 2009 23:19:12 +0000

I finally got around to verifying my websites with the Google, Yahoo! and MSN Live search engines. I’ve also setup sitemaps for my websites too. I’ve done this for the websites at work but couldn’t be arsed to do it for my own sites until today.

To find out why you want sitemap enable your sites read the Sitemap page at Wikipedia. In order to register your sitemaps with the major search engines you’ll need to setup accounts for Google Webmaster Tools, MSN Live Webmaster Tools and Yahoo! Site Explorer.

You’ll then need to verify your site(s) with each of the search engines, they provide details about how to do this but it typically requires that you put a meta tag in your page header. Once you’ve verified your site(s) it’s time to create a sitemap.xml.

If you have a Wordpress blog adding sitemap support is easy, just use the Google XML Sitemaps plugin. This re-builds sitemap.xml when you update posts, pages, comments etc. It also notifies all the relevant search engines that content has changed on your site. Very useful indeed. If you need to create your sitemap by hand, or better yet script its creation, then read sitemaps.org for the protocol spec.

Blog post notifications to Twitter

martin@wimpress.com (Martin Wimpress) — Sat, 17 Jan 2009 10:45:12 +0000

I’ve just installed WordTwit on my blog. WordTwit keeps track of when you publish new posts, and automatically informs all of your followers by pushing out a Twitter tweet. All links are automatically converted to tiny URLs.

SSHMenu - SSH Connection Management

martin@wimpress.com (Martin Wimpress) — Tue, 26 Feb 2008 13:21:28 +0000

We have a reasonable number of Debian servers at work and as a result I ssh into servers about as many times as I visit Google. I have been using Profiles in gnome-terminal to manage my ssh connections, which is fine but requires I already have a terminal open to initiate a new server connection. Enter SSHMenu, a GNOME panel applet that keeps all your regular SSH connections within a single mouse click.

I couldn’t be arsed adding up a new repo for one application, so here are my quick and dirty install steps.

wget http://sshmenu.sourceforge.net/debian/dists/stable/contrib/binary-all/sshmenu_3.15-1_all.deb
wget http://sshmenu.sourceforge.net/debian/dists/stable/contrib/binary-all/sshmenu-gnome_3.15-1_all.deb
sudo gdebi -n sshmenu_3.15-1_all.deb
sudo gdebi -n sshmenu-gnome_3.15-1_all.deb

I have taken to using gdebi to install local deb packages as it resolves and installs dependencies. Now add SSHMenu to a GNOME panel and configure your ssh connections. If SSHMenu isn’t listed in the GNOME panel applets yet, then you can force a refresh with the rather heavy handed…

killall gnome-panel

Get some AIR

martin@wimpress.com (Martin Wimpress) — Tue, 12 Feb 2008 12:53:03 +0000

AIR (Automated Image and Restore) is a GUI front-end to dd and dcfldd designed for easily creating forensic bit images. Or, a nice way to let the guys at work who like GUIs make Debian boot install floppies (don’t ask) easily. Ubuntu 7.10 doesn’t have a package for AIR, but AIR does have an installer.

sudo aptitude install perl-tk sharutils dcfldd netcat cryptcat
wget http://prdownloads.sourceforge.net/air-imager/air-1.2.8.tar.gz?download
tar zxvf air-1.2.8.tar.gz
cd air-1.2.8/
sudo ./install-air-1.2.8

At this point the installer thingy will kick off, follow the prompts. I have found that air needs to run as root, but won’t run if the executable is setuid. So run it via sudo like so:

sudo air

gLabels 2.2.1

martin@wimpress.com (Martin Wimpress) — Wed, 30 Jan 2008 11:45:57 +0000

gLabels, as packaged in the Ubuntu repositories, has not worked properly for some time. The accuracy of printing was way out making gLabels a non-starter unless you went to the hassle of manually re-aligning every label on a page to account for the inaccuracies.

However, I have been patiently waiting for a new version of gLables to be released. The new development branch completely replaces libgnomeprint with the new GtkPrintOperation and Cairo. The upshot of that is that the printing accuracy issues are resolved.

GetDeb have released .debs for gLabels 2.2.1 that work with Ubuntu Gutsy 7.10. You can either download the .debs from the gLabels page at GetDeb and let gdebi do its thing or do the following from the shell…

wget -c ftp://cesium.di.uminho.pt/pub/getdeb/gl/glabels_2.2.1-1~getdeb1_i386.deb
wget -c ftp://cesium.di.uminho.pt/pub/getdeb/gl/glabels-data_2.2.1-1~getdeb1_all.deb
sudo dpkg -i glabels*.deb

We print a lot of labels at work to identify media for flight recorders, that job just got a whole lot easier. More importantly, I now have a viable address label printing solution for my wife.

BloGTK, a weblog client for Linux

martin@wimpress.com (Martin Wimpress) — Fri, 11 Jan 2008 13:44:07 +0000

I have been meaning to setup a weblog client for a while now. I have tested a couple of blog clients and have settled on BloGTK.

It has a simple user interface but comprehensive features, although I did need to define a few Custom Tags before the editor supported all the formatting options I wanted. Setting up BloGTK is very simple for Ubuntu and Debain users requiring an aptitude install blogtk to get it installed and the following settings will connect to a Wordpress blog.

Server URL: http://blog.example.org/xmlrpc.php
Account: Your Username
Password: Your Password
Blogging System: Moveable Type

Now I can blog directly from my desktop, I am hoping it will encourage me to post more often.

Zindus contact sync for Thunderbird and Zimbra

martin@wimpress.com (Martin Wimpress) — Thu, 10 Jan 2008 15:41:24 +0000

When I setup the mail server at work I picked Zimbra. It’s great.

Today Zimbra just got a bit better when I found Zindus. Zindus is open source software and runs on all Thunderbird platforms including Windows, Mac OSX and Linux. Zindus can sync your contacts between Zimbra and Thunderbird. It syncs everything from Address Books to your GAL (Global Address List).

It currently supports ZCS 3.x to 5.x. Adding the Lightning and Zindus add-ons to Thunderbird create a very complete desktop client to Zimbra for any platform. Can’t wait to share to good news with my Outlook users.

Simple iCal Server

martin@wimpress.com (Martin Wimpress) — Wed, 09 Jan 2008 23:20:28 +0000

For the longest time I have been meaning to setup a shared calendar for my wife I and to use. You see, like most men I have no idea when and where I am supposed to be. This is because my wife keeps all this information in her filofax and that lives in her handbag, somewhere I never venture. So I have spent this evening setting up PHP iCalendar on Lighttpd, and I am very happy with the results.

I have opted to use the publish.php add-on provided with PHP iCalendar, rather than add the WebDAV module to Lighty. For our modest requirements it works very well. We now have full read/write access to our calendar by using the Lightning extension for Thunderbird and read only access via the PHP iCalendar web interface. Viewing the calendar via the web interface requires a login first and the calendar publishing is protected by Lighty authentication using htdigest.

Lastly, because our calendar is a nice open standard I have options as to how I might sync it with my mobile phone. More on that when I figure it out.

Flexion.Org is re-activated!

martin@wimpress.com (Martin Wimpress) — Sat, 22 Sep 2007 11:22:43 +0000

I have finally got around to re-activating this website after many months of neglect. The first thing to do was move away from Jaws. Sadly, my Jaws installation is broken in a way that newer Jaws releases refuse to upgrade it.

I did some research and decided to migrate to Wordpress since it caters for all the essentials I require either in the core functionality or via plugins. Plus I can integrate with some other stuff I want to use.

The migration to Wordpress was made all the easier thanks to the excellent Jaws import plugin I found on the Wordpress trac. However, I decided not to use the Textile plugin to preserve the Jaws formatted text. Instead I chose to manually clean up some of the old entries so I could stick with the TinyMCE editor long term.

I am happy with Wordpress thus far and look forward to getting some plugins going and picking up a new theme over the coming days and weeks.

POLARIS

martin@wimpress.com (Martin Wimpress) — Tue, 01 May 2007 00:00:00 +0000

Led the design, development and operation of the POLARIS project at Flight Data Services. An Open Source analysis engine developed using Python that can scale to analyse and report on millions of commercial aviation flights to comply with global aviation safety regulations. Now known as Flight Data Connect and owned by L3HARRIS and apparently no longer available as Open Source.

Organisation: Flight Data Services
Date: May 2007 - September 2015
Role: Manager of IT Operations & Software Engineering

R3-born 0.1.1 is released!

martin@wimpress.com (Martin Wimpress) — Fri, 09 Mar 2007 21:59:45 +0000

R3-born 0.1.1 is released :-)

R3-born Core Change Log

Here are the main changes for this release. v0.1.1 : 9th March 2007

Added Akismet anti-spam API
Added a version checker for the Core and also for Blocks and Modules.
Added search bot (Google, MSN, Yahoo!, etc) detection.
Added logging (in Combined Log Format) to the R3-born Intrusion Prevention System.
Enhanced the Module API so it now supports safe updating and custom CSS for blocks and modules.
Enhanced the Layout Manager
Enhanced the Registration and Profile pages by separating them.
Enhanced the Template engine by back porting some features from phpBB 3.0.x
Optimised and fixed the Search API
Optimised and simplified Coloured Groups.
Improved the Groups Management ACP
Improved the Styles Management ACP
Fixed various bugs in the core.
Significant feature enhancements, bug fixes and optimisations added to the Menu and Who Is Online blocks.
Significant feature enhancements, bug fixes and optimisations added to the Comments, Downloads, Forum, Pages,
Private Messages, Weblogs and View Online modules.
New modules : bot_list, cookie, error, google, referrers and sql_backup
New blocks : recent_blogs, recent_comments, recent_pages, recent_topics, search, socmarks and visit_counter.

R3-born 0.1.1 has fulfilled its first major objective, which is to be functional enough to run a website.

R3-born 0.1.0 is released!

martin@wimpress.com (Martin Wimpress) — Tue, 30 Jan 2007 11:52:55 +0000

Finally, R3-born 0.1.0 is released. R3-born is a content management system for modest websites which I hope will be ideal for individuals running their own sites or small community websites.

Due to my accident I have not been able to address all of the bugs which have been reported recently and the downloads are only available as bzip2 tarballs right now. Please report any bugs you might encounter as 0.1.0 has really only been tested on Debian Sarge (stable) running PHP4 and MSQL 4. Feedback from other configurations will be useful and all reported bugs will be investigated.

Announcing R3-born

martin@wimpress.com (Martin Wimpress) — Fri, 08 Sep 2006 17:48:47 +0000

I suppose it is a bit silly to announce this just before I go on holiday but what the heck. R3-born is intended to be a simple and modular content management system for modest websites. My hope is that it will be ideal for individuals running their own sites or for small community websites. R3-born is coded in PHP (4.2.0 or better required) and requires an SQL database, currently MySQL 3.x, 4.x and 5.x, Postgres 7.x or better and MS-SQL 7.x or better are supported.

Current Status

Right now, R3-born is considered pre-alpha and not suitable for a production website but is ready for testers to start feeding back on bugs, feature requests and in the near future code changes. Development has been fairly rapid so far. I started the project in April 2006 and I hope to release a beta version of R3-born later in 2006.

http://www.sourceforge.net/projects/r3-born/

Conslidated RavenCore HOW-TOs

martin@wimpress.com (Martin Wimpress) — Fri, 18 Aug 2006 12:37:07 +0000

I have reorganized and consolidated my Debian and Ubuntu HOW-TOs for RavenCore 0.2.3.

RavenCore 0.2.3 on Debian and Ubuntu

martin@wimpress.com (Martin Wimpress) — Thu, 17 Aug 2006 10:53:53 +0000

I have updated my HOW-TO for installing RavenCore on Debian (at the request of the RavenCore developer) and added a HOW-TO for installing RavenCore on Ubuntu. The are still some minor compatibility issues with Debian and Ubuntu and these HOW-TOs have been sent to the RavenCore developer to help document the outstanding incompatibilities.

Comments off, Overdue update and R3-born

martin@wimpress.com (Martin Wimpress) — Tue, 04 Jul 2006 13:14:55 +0000

First of all, I have disabled comments on my blog. Jaws has spam protection and captachs but both implementations suck. This site doesn’t generate much feedback so disabling comments altogether is not really a problem. I haven’t updated the site recently and here is why. The 0.6.x release of Jaws is buggy and I still haven’t found anything else which fully meets my needs for managing my own web content.

Therefore for the last couple of months I have been working on an open source CMS of my own called R3-born. More details to follow about that soon, but you can find the project over at Sourceforge, don’t expect anything to work (although it does for the most part) it is pre-alpha and unreleased. SVN is the only way to snag the source.

R3-born

RavenCore 0.1.5 on Debian

martin@wimpress.com (Martin Wimpress) — Sat, 29 Apr 2006 19:18:15 +0000

I have updated my HOW-TO for installing RavenCore 0.1.5 on Debian. I’m pleased to report that the RavenCore developer has been very responsive to my feedback and has significantly improved “out of the box” Debian compatibility in this release. The are still some minor compatibility issues and I’ve’ sent feedback to the developer in the hope that things will continue to improve with RavenCore 0.1.6.

Project Minerva is coming home

martin@wimpress.com (Martin Wimpress) — Wed, 26 Apr 2006 17:42:11 +0000

Despite a generous offer from a Minerva enthusiast a year or so ago to freely host the Project Minerva sites, all the Project Minerva hosting is coming back to where it started and rightly belongs. Yes, right here with me. It just goes to show you really don’t get anything for free. The developers have had several outages with the server which was provided by SpiderTech Hosting, so we are bringing it back here where we can look after our interests directly.

Minerva users might be wondering if the fact I am fully hosting Project Minerva again means I am going to be developing for the project again. Well, we will just have to see but I did inform Chris of a little project I kicked off last week which may (or may not) lead to something.

Today I have been mostly lovin' Munin

martin@wimpress.com (Martin Wimpress) — Fri, 07 Apr 2006 17:00:00 +0000

Here’s the blurb from the authors… Munin surveys all your computers and remembers what it saw. It presents all the information in in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort. Using Munin you can easily monitor the performance of your computers, networks, SANs, and quite possibly applications as well. It makes it easy to determine “what’s different today” when a performance problem crops up. It makes it easy to see how you’re doing capacity wise on all limited resources.

I am a winner!

martin@wimpress.com (Martin Wimpress) — Tue, 04 Apr 2006 16:00:00 +0000

Yah for me! I have won the RavenCore documentation contest.

Flexion.Org Server Migrated

martin@wimpress.com (Martin Wimpress) — Tue, 04 Apr 2006 15:20:29 +0000

Today Flexion.Org was successfully migrated to a new server. I’ve been using the same shared host for over 7 years but these days I really need much better control over my server, mostly for spam/virus protection of e-mail but also to ensure that periodic upgrades don’t screw up my website.

So this time I opted for a virtual dedicated server and I had a choice of all the popular Linux distros, which I can re-install via a web interface at will in a few seconds. I choose. If you are looking for a low cost virtual dedicated server then take a look at Quantact.

RavenCore 0.1.4 on Debian

martin@wimpress.com (Martin Wimpress) — Wed, 22 Mar 2006 18:02:05 +0000

I set up my new server to use RavenCore so that managing my domains and e-mail accounts for friends and family will be as easy as possible. RavenCore is designed for use on CentOS primarily and therefore RavenCore 0.1.4 requires some tweaking in order to get it fully working and compatible with Debian.

I have been sharing my feedback with the RavenCore author and when RavenCore 0.1.5 is released “out of the box” Debian compatibility should be a reality.

Portable app

martin@wimpress.com (Martin Wimpress) — Thu, 09 Mar 2006 05:40:07 +0000

I started a new job last week, I am contracting for the first time. I needed some applications installed on my desktop at work, but to do so would not be permitted as part of my clients standard build policies.

So I started making a USB disk with the tools I needed and in doing do I found the Portableapp and The Portable Freeware Collection websites which ease the process no end.

A portable app is a computer program that you can carry around with you on a portable device and use on any Windows computer. When your USB thumb drive, portable hard drive, iPod or other portable device is plugged in, you have access to your software and personal data just as you would on your own PC. And when you unplug, none of your personal data is left behind.

Slimlining Windows XP with nLite and XPLite

martin@wimpress.com (Martin Wimpress) — Sun, 05 Mar 2006 06:41:38 +0000

A shocking truth is that despite my preference for Linux I do still have to use Microsoft Windows/ I have recently started a new job where I am now required to use Windows XP on my desktop at work and I also need a Windows computer at home in order to update firmware/software and configure most devices in my home cinema.

So I have decided to install Windows XP on my most aging of laptops so it is always available for my home cinema tweaking, this also means I can remove dual boot Windows partitions on my main computer. However, because my old laptop is a bit short on memory I need to tweak Windows XP a good deal. I have been a long time user of 98Lite and XPLite from LitePC.om and have been slim lining Windows for many years. I have recently found nLite which does much the same as XPLite except it rebuilds the Windows XP (or Windows 2000 and Windows 2003) installation CD so that the configuration changes you make, and Windows components you remove, are ingrained in a new ISO image.

The biggest benefit is that using nLite I can integrate Service Pack 2 and all the current hotfixes and patches into the new ISO meaning that after an install the new Windows system is better secured. I have also been able to integrate Firefox into my custom Windows XP install CD and completely remove Internet Explorer, and by using the WindizUpdate it is possible to do online Windows Updates using Firefox, Opera and K-Meleon.

There are also many other benefits to using nLite other than simply removing or adding software components, you can also pre-configure just about every aspect of the Windows environment and nLite will wrap all this into an unattended installation ISO image. My custom Windows XP install CD I made with nLite is just 145mb!

NetworkManager and Ubuntu Dapper Drake 6.04

martin@wimpress.com (Martin Wimpress) — Thu, 02 Mar 2006 11:26:39 +0000

A week or so ago the NetworkManager package was changed so that nm-applet (the system tray application) was separated into it’s own package. After I updated and installed nm-applet, I noticed that it didn’t startup anymore. I didn’t have time to investigate until this morning and found the answer in the following forum posting…

NetworkManager broken after last update

Here is the pearl of wisdom, from that forum discussion, that helped me make my system to be NetworkManager compatible again.

“Network Manager has been installed on your system, however it will not immediately be able to manage your network interfaces. To avoid problems with important configuration being ignored, or strange behaviours, the Ubuntu version will not manage any network interface configured in the /etc/network/interfaces file.

To allow interfaces to be managed with Network Manager either edit the /etc/network/interfaces file and remove (or comment out) the “auto” and “iface” lines for those interfaces you wish it to manage, or use the “Networking” administration tool (found under the “System” menu) to disable the interfaces.”

Long weekend in Dublin

martin@wimpress.com (Martin Wimpress) — Mon, 27 Feb 2006 09:57:32 +0000

We just got back from an enjoyable trip to Dublin, highlights for me were visits to the…

Guinness Storehouse for a history lesson and a drop of the black stuff.
The Old Jameson Distillery for another history lesson and a drop of the hard stuff, I also certified as an Irish Whiskey taster during the visit. I have a certificate and everything :-)
Gallagher’s Boxty House for some traditional Irish grub.
Ryanair’s ticket prices, just 10p return each!

This was our first visit to Ireland and I am looking forward to going back again soon and maybe incorporate some diving.

DVD Box Sets - Extended Film Watching Session is Coming

martin@wimpress.com (Martin Wimpress) — Wed, 15 Feb 2006 12:25:47 +0000

Today I finally got around to ordering some stuff from Amazon.co.uk using the vouchers I was given as present at Christmas. I have ordered The Matrix Trilogy and The Lord of the Rings Trilogy (Extended Edition Box Set) which I hope arrive before the weekend as my wife is away for a few days which should be enough time to get through both box sets 😄

New Mountain Bike

martin@wimpress.com (Martin Wimpress) — Tue, 14 Feb 2006 12:49:30 +0000

I finally convinced my wife that I need (really wanted) a new mountain bike and today I took delivery of the Trek 4300 Disc 2006 model

I managed to fall off it on the maiden voyage but I thought it best to get that out that way before I tackle to Bracknell Woods this coming Sunday.

Cheap and Cheerful MP3 Player

martin@wimpress.com (Martin Wimpress) — Fri, 10 Feb 2006 13:00:10 +0000

My six year old MP3 player just isn’t up to the job anymore and I have been looking for an alternative, on and off, over that last few weeks. Ideally I would have liked a portable music player that supported Ogg Vorbis but as my home cinema isn’t Ogg Vorbis compatible I decided to standardize on MP3. No flames please!

I have lots of MMC and SD memory cards that I have collected over the years so decided get a player which could make use of them. I only need a player for use in the gym and while cycling so it doesn’t need to be too fancy, with that in mind I decided to get the Sumvision XT105 MP3 Player which is super cheap and surprisingly good considering the price :-)

Initial impressions are that the manual was obviously translated from Chinese to English using Google, the included headphones are not much cop. That said, once you plug-in a decent set of head phones the sound reproduction is very good and more than good enough for what I want it for and although playlists aren’t supported playback by folder is. That player is very light and very small, so ideal for use in the gym and battery life is around 10-12 hours from a single AAA battery. It is also Linux compatible, although I have been using a USB card reader to populate my memory cards with MP3s as the XT105 is only USB 1.1 and therefore a bit slow. I am off to the gym now with some banging tunes.

Panasonic Plasma Black Level Fluctuation Fix

martin@wimpress.com (Martin Wimpress) — Fri, 10 Feb 2006 02:48:54 +0000

I’ve noticed that the contrast levels on my Panasonic TH42-PHD8 plasma will suddenly increase during dark scenes and then drop back down to normal a fraction of a second later. This can be most distracting but as it only happens once or twice a week it hasn’t really bothered me. But, I found a discussion at AVForums which describes how to enter the plasma service menu and fix this behaviour. I’m pleased to say it works perfectly. Here’s the solution:

Panasonic Black Shift Problem/Solution

The honeymoon is over

martin@wimpress.com (Martin Wimpress) — Fri, 03 Feb 2006 11:43:06 +0000

Despite my glowing comments about Fedora a couple of weeks back, the honeymoon period is well and truly over. My biggest gripe with Fedora is even with numerous compatible repositories enabled there isn’t a suitable selection of packages available and dependency breakage is all too frequent.

Although I can build my own packages from source I just don’t have the time (or inclination) for that anymore. I do really like the Early Logon feature and the working NetworkManager in Fedora, but sadly these features are not enough to keep me away from Ubuntu. As a result I have just finished migrating all my computers to Ubuntu Dapper Drake 6.04 - Flight 3, which is shaping up to be another excellent release. Matt Galvin has been doing a great job of tracking the new features in each of the Dapper Flight pre-releases, see the Wiki entries below…

Having spent 6 weeks using Fedora I have decided that I have learnt enough about it to conclude it just isn’t for me and I will be seeing out the rest of 2006 using Ubuntu as my desktop OS. However, I will dabble with other distros periodically.

Leading edge Fedora Core 4

martin@wimpress.com (Martin Wimpress) — Sat, 14 Jan 2006 11:32:31 +0000

Having spent all of 2005 using Ubuntu I have learnt a lot about Debian and even started using Debian for my servers.

Despite using Linux for over 10 years I had never previously used Debian and had even avoided using it. I was surprised by how much I liked using Ubuntu and Debian and decided that I give Fedora another try as I have never warmed to redhat in the same way I had never previously warmed to Debian. For the longest time I used Slackware and then switched to Crux.

This week I have installed Fedora Core 4 on my home computer and been hugely impressed, but this is mostly down to finding the nrpms.net repository. I was already using the RPMforge.net group of repositories and nrpms.net is striving for RPMforge compatibility, so far I haven’t experienced any package or dependency breakage so they must be doing something right. Thanks to nrpms.net I have effortlessly upgraded FC4 to GNOME 2.12.1, Firefox 1.5, mono 1.1.12 plus numerous other GNOME applications I regularly use which aren’t available elsewhere. The real bonus was to get NetworkManager 0.5.1 from nrpms.net which actually works, and works reliably!

As a result, this is a Linux distribution I can install on my wifes laptop without fear of reprisals. I have found that installing proprietary hardware drivers and software has been much simpler for Fedora than Ubuntu, most notable are VMWare Player, Linuxant HSF modem drivers, ATI drivers, Opera and Skype.

I have been able to get my Smartphone and PocketPC working under Ubuntu the amount of tweaking and constant manual intervention is cumbersome. So I was staggered when after installing the SynCE packages on Fedora both devices just worked when connected via USB, the whole process was completely seamless.

So, after a few days tinkering I have everything I need installed, configured and working reliably. Fedora is not perfect thought, yum is still lagging behind apt and even that even after enabling a number of the excellent 3rd party repositories the number and variety of packages available to FC4 isn’t that great. Installing software is just so slow, what the bloody hell is yum doing? Using yumex is coma inducing and up2date just doesn’t work.

That said, there is some work being done in these areas for FC5 so maybe the gap will be closed on apt a little in the coming months. I do like the fact that the official Fedora repositories update packages past the final release date, something which Ubuntu doesn’t do. This means that on my first yum update OpenOffice was upgrade from 1.9.140 Beta to 2.0.1 final and many other essential applications were also upgraded. Just when I thought it couldn’t get any better I discovered the “Early Login” feature which is new to Fedora Core 4. It is simply brilliant and I can not get from power button to GNOME desktop in 20 seconds. I so wanted to not like Fedora but I have to say I think I might be converted. I will have to see how long this enthusiasm for Fedora will last, but I am already looking forward to FC5.

Powerbook G3 (Old World) Ubuntu Install

martin@wimpress.com (Martin Wimpress) — Mon, 09 Jan 2006 11:28:54 +0000

James bought a PowerBook G3 to the January 2006 HantsLUG meeting and wanted to have Ubuntu installed on it. I used to be familiar with Apple Mac computers in the mid to late 90’s because the company I worked for used Macs exclusively on the desktop, so I decided to give it a whirl.

The bottom line is that this old PowerBook G3 is now running Ubuntu Hoary 5.04 quite happily and I will soon be phone James to arrange for him to pick up his computer.

The article is largely based on the OldWorldsMac Ubuntu Wiki, I have included the specifics of James setup here so he has a record of how his computer was configured.

I quickly realised that simply inserting the Ubuntu Hoary 5.04 PowerPC install CD and rebooting while holding down the ‘C’ key wasn’t going to work. It seems that James computer is old enough to be a right pain in the arse to get working with Linux, but that just makes me want to get it working all the more :-)

By the end of the HantsLUG meeting Ubuntu Hoary 5.04 was just completing the final setup steps and would have been ready to use, but someone pulled the power block out of the mains and James battery is dead, so the computer shutdown :-( I took the PowerBook G3 home with me to fix/complete the installation but when I got home I decided to download Ubuntu Breezy and install that instead. Sadly, James PowerBook only has a 2GB hard disk (1.9GB available for Linux) and this wasn’t sufficient for the Breezy installation to complete so I reverted to Hoary.

Identifying the PowerBook G3

After some Googling I soon discovered that there are two classes of PowerBook G3, OldWorld and NewWorld, and that each of the classes have a number of slightly different models in the range. I took a guess that James was an OldWorld PowerBook based on the fact it had no USB ports. It turns out this was a good guess.

Making the assumption that James PowerBook was OldWorld meant that I should use BootX to boot into Linux from MacOS as opposed to using yaboot which is for NewWorld PowerPCs.

PowerBook G3 Computers: How to Identify Different Models

Doing the MacOS Installation

The first job was to erase the hard disk and install MacOS in a small parition and leave the rest of the drive unallocated for installing Ubuntu later on.

Insert MacOS 8.1 CD
Reboot
Hold down ‘C’ as the computer boots to get it to boot from CD
Erase the Hard Disk
Use Drive Setup to make a 150MB HFS partition.
Install MacOS, leaving off all the optional components. The only thing you need is Stuffit and a browser which are both part of the minimal MacOS8.1 install.

After the install I delete some odds and ends I didn’t need and also used Extensions Manager to deselect a lot of components that were not required. Finally, I configured TCP/IP to use DHCP.

Tools Update

James wanted to be able to use MacOS for basic web surfing, so that meant finding an more modern alternative to Netscape 3.0.3 which was installed be default. Sadly The only (relatively) upto date browser I could find which would install on MacOS 8.1 was Internet Explorer 5.1.7. I also needed an updated version of Stuffit Expander to extract BootX 1.2.2. I used Netscape 3.0.3 to download IE and Stuffit from the URL’s below…

…and installed them. Once they were working a deleted Netscpae 3.0.3 and the old version of Stuffit.

BootX

Older versions of Stuffit will only extract BootX 1.1.3 which I why I updated Stuffit Expander as explained earlier. Download BootX 1.2.2.

BootX

Open it with Stuffit and extract it the the Desktop then open the resulting BootX 1.2.2 folder on the desktop. Drag each of the following…

BootX
BootX Extension
Linux Kernels

…onto the “System Folder” one at a time to install BootX correctly. Insert the Ubuntu PPC install CD and navigate to the /install/powerpc folder.

Copy vmlinux to (the Linux kernel) System Folder/Linux Kernels
Copy initrd.gz (the init ramdisk image) to System Folder/ and rename it to ramdisk.image.gz

BootX should appear on the apple menu and also run on every reboot during the boot process, meaning that you can choose to boot Linux without having to wait for the entire MacOS to load :-) When you run BootX it should show ‘vmlinux’ as an available kernel, now add the following to “More kernel arguments” to make sure the correct video mode is used for Linux.

video=atyfb:vmode:14,cmode:32,mclk:71

Now click the “Options” button, check “Use Specified RAM disk” and select System Folder/ramdisk.image.gz. Click on the “Save to prefs” button and then click on the “Linux” button and in a short while you should be looking at the regular Ubuntu install dialogs.

Doing the Ubuntu Installation

The installer will display an error message that “Configure a multiseat system” failed. You can ignore this…

Select ‘Continue’
Select ‘Detect Hardware’ and press Enter

…and the install will continue normally.

Partitioning

When it gets to partitioning the drive Ubuntu will suggest using the entire disk for Linux. Don’t do that because you still need MacOS to run BootX to bootstrap Linux. Select the “Use Free Space” option or partition manually.

Copying /boot to the HFS System Folder

The rest of the install is fairly straight forward until you get to the part where Ubuntu tries to install a bootloader. GRUB and Lilo don’t work on OldWorld Macs, so Ubuntu will warn you that no bootloader can be installed. Switch to a second console at this point (Option-F2) and use df to see where things are currently mounted. In my case the newly installed ubuntu was on /dev/hda8 mounted as /target and the HFS filesystem was /dev/hda7. Make a mountpoint and mount the HFS filesystem.

cd /target
mkdir hfs
mount /dev/hda7 hfs -t hfs

You might also want to add an entry to /etc/fstab so it will be mounted when you reboot. This makes updating kernels easier in the future.

echo '/dev/hda7 /hfs hfs defaults' >> etc/fstab

Now copy the kernel and boot image over;

cp boot/vmlinux hfs/System\ Folder/Linux\ Kernels/vmlinux
cp boot/initrd.img hfs/System\ Folder/ramdisk.image.gz

Option-F1 to get back to the installer, and tell it to go ahead and reboot.

When the machine reboots the BootX dialog should come up straight away, just click “Linux” and Ubuntu should proceed through the rest of the install as usual.

References

RavenCore 0.0.6 on Debian

martin@wimpress.com (Martin Wimpress) — Thu, 05 Jan 2006 11:24:41 +0000

I have been testing a few virtual hosting control panels the last couple of days and I am deeply impressed with RavenCore, so much so I will most likely use it for a project I am planning. However, Debian is my server distro of choice and RavenCore needs a little tweaking in order to work seamlessly with Debian. I have been keeping some notes that explain how to get RavenCore to play nicely with Debian.

I have submitted my notes in the RavenCore Forums and agreed to assist the author with any future testing to work toward better “out of the box” Debian integration.

Ubuntu

martin@wimpress.com (Martin Wimpress) — Sun, 01 Jan 2006 00:00:00 +0000

Ubuntu comes with everything you need to run your organisation, school, home or enterprise. All the essential applications, like an office suite, browsers, email and media apps come pre-installed.

Community contributor since 2006, worked for Canonical from 2016 to 2021 progressing from software engineer to director of engineering. I remain active in the Ubuntu community.

Organisation: Ubuntu Project & Canonical
Date: January 2006 - date
Role: Community contributor

Ubuntu Dapper Flight-2

martin@wimpress.com (Martin Wimpress) — Thu, 22 Dec 2005 11:23:01 +0000

I decided to install Ubuntu Dapper Flight-2 on a spare computer I had available. There are not any obvious differences between Dapper and Breezy at this early stage but one thing which does stand out is the performance improvements. My test Dapper and stable Breezy computers are exactly the same specification and the boot times for Dapper are vastly improved, currently 37 seconds (average) faster from power on to GNOME desktop than with Breezy. The graphical boot menu when booting from the install CD in a nice touch, I look forward to seeing how Dapper improves over the coming months. But right now there isn’t anything especially new, different or unusual to test.

Ubuntu Backports and PLF Repositories

martin@wimpress.com (Martin Wimpress) — Wed, 21 Dec 2005 11:19:04 +0000

I haven’t been focusing on Linux during that last couple of months, I was a bit distracted completing my home cinema installation and ever since I have been watching a lot of films.

However, I needed to get one of my laptops up to date for the Christmas holidays and in doing so needed some packages from the Ubuntu Backports repository, which led me to find the Ubuntu Penguin Liberation Front.

The Ubuntu PLF is a team that builds packages that are patent encumbered or proprietary. The PLF has been providing litigious packages for Mandriva for many years and now they are doing the same thing for Ubuntu. Having added the Breezy Backports and PLF repositories (just about) everything I consider essential for my desktop needs is now apt-getable, yah!

PaceTwin USB Interface Modification

martin@wimpress.com (Martin Wimpress) — Sat, 17 Dec 2005 11:17:14 +0000

I had previously read on Yahoo Groups that a project was under way to retro fit a USB interface to the PaceTwin PVR allowing for easy archiving of recorded programs to a computer. Well, the guy who is planning the DIY upgrade has setup a website which is documenting progress and also other useful PaceTwin information. See the URL below for more information…

http://www.pace-twin.org.uk

Lumagen Video Processor Bug Fixed

martin@wimpress.com (Martin Wimpress) — Mon, 12 Dec 2005 12:10:33 +0000

I have been exchanging e-mails with the UK distributor for Lumagen and the Lumagen engineering department. Today is a happy day because I have just tested a pre-release firmware which fully fixes the issues I’ve reported.

Thanks very much to the engineers at Lumagen amd for their patience and hard work. Truly excellent customer service! See my post in the Lumagen forums.

UK DVB-T PAL De-interlacing Issues

Lumagen Video Processor Bug Hunting

martin@wimpress.com (Martin Wimpress) — Tue, 22 Nov 2005 12:02:37 +0000

I have spent the last few weeks tweaking and improving my home cinema setup, most of my time has been spent learning how to fully exploit my Lumagen video processor. I am extremely impressed with the huge improvements in image quality the Lumagen VisionDVI provides as opposed to sending video sources direct to the plasma screen, but something isn’t right with the de-interlacing of DVB-T (or Freeview as we call it in the UK). I have isolated the exact issue and posted a problem report in the Lumagen forums.

Home Cinema is complete!

martin@wimpress.com (Martin Wimpress) — Sat, 05 Nov 2005 11:52:46 +0000

Tonight was a landmark occasion. After nearly two years of planning I finally completed our home cinema. The last job was mounting the plasma screen. Apart from hiding away some power cables it is complete and I can’t wait to get into some serious film viewing. Mounting the plasma screen was not a job I was looking forward as DIY is not my strong suit. But, I managed it and posted my experiences in the topic below at AVForums.

Mounting on dry-lined walls

First Prototype of my Linux PVR

martin@wimpress.com (Martin Wimpress) — Tue, 11 Oct 2005 11:05:06 +0000

I have built an initial prototype of a Linux based PVR using some (very) old hardware I had available. I simply plugged in a new Hauppauge Nova-T PCI card and hoped for the best. I got everything working but had to fudge things so that over the air electronic program guide (EPG) can be imported into the MythTV program guide database.

This initial prototype uses KnoppMyth Release 5A16 so that I could get up to speed as quickly as possible but I plan on using Debian or Ubuntu in the future.

When Linux meets Home Cinema

martin@wimpress.com (Martin Wimpress) — Wed, 05 Oct 2005 11:02:43 +0000

I have started working on a new project at home, it is one I know I am going to enjoy because it satisfies two of my hobbies. I am going to build a prototype PVR based on MythTV.

Isles of Scilly

martin@wimpress.com (Martin Wimpress) — Wed, 21 Sep 2005 10:31:09 +0000

I arranged a surprise holiday to the Isles of Scilly for my wifes birthday and we got back a couple of days ago. What a fantastic place!

We very much enjoyed our short time on the islands and will definately be going back in the future, I think our enjoyment was also heightened by the fantastic weather we had during our stay. One of the highlights of the trip was the helicopter flights to and from the islands, which gives you some great views of the Cornish coast line and Lands End. The Scilly Isles have a rich history and this is slowly revealed as you visit each island and the attractions they offer.

The scenery is beautiful everywhere, but the Abbey Garden on Tresco is very impressive. A side from island hoping and wandering around the island coastal pathways we also went out to see some grey seals out on the rocks to the west of the islands. Other highlights are the plentiful supply of real Cornish pasties, cream teas and ice cream. The food on the islands was very good and if you like fish you are in for a treat. The views of the sunsets are lovely and best accompanied with a pint of ale :-)

Handheld Linux

martin@wimpress.com (Martin Wimpress) — Mon, 12 Sep 2005 10:27:05 +0000

I have replaced the disk in my Libretto CT100 with a 20GB disk I had going spare and re-installed Debian Sarge 3.1, this time I didn’t bother with the 2.6 kernel or Xfce because I’ve had my fun with those now. But playing around with Linux on my Libretto has re-kindled my interested in running Linux on small portable devices.

Earlier this year I experimented with Familiar Linux 0.8.2 on my iPAQ 3970 and installed via a serial cable, which was very slow going. So I’ve invested in a 512MB Compact Flash card to speed up my next round of tinkering with Familiar but this time I intend to leave Familiar installed and not re-flash the 3970 with PocketPC 2002. The Familiar v0.8.2 Installation guide covers pretty much everything you need to know about the installation process.

Debian 3.1 on Libretto CT100

martin@wimpress.com (Martin Wimpress) — Fri, 09 Sep 2005 10:26:14 +0000

I decided to stick with the 2.1GB hard disk in the end, but in order to complete the initial install I fitted it in an old IBM Thinkpad 600X. This allowed me to boot off the Debian 3.1 Net Install CD and get a basic system installed. Once that was done I fitted the hard disk back in the Libretto and installed the essential bits and pieces I needed. I am feeling a lot of love for apt-get today.

Then I got carried away. I upgraded to the 2.6 kernel, installed X and Xfce 4.2.2. Then I migrated to the testing repositories, the upgrade for which has finished a few minutes ago. I found Quentin Stafford-Fraser’s Linux on the Libretto 100CT page very helpful in finding the right Modeline settings to get X going. I don’t really need Xfce on the Libretto but I couldn’t resist getting it running. Despite the low spec processor and just 64MB of RAM it runs very well, even Firefox is usable. I might have to bring this box along to the next HantsLUG meeting.

Server crash! But I have the perfect replacement

martin@wimpress.com (Martin Wimpress) — Fri, 09 Sep 2005 10:18:22 +0000

The CPU fan on my home server stopped spinning, which has rendered the computer unusable until I find the time to replace it. I have my workstation, so I can keep working for the most part, but the services I was running on the server are critical so I need a server replacement ASAP. I have decided to setup a dedicated server for those services.

I always knew my Libretto CT100 would come in handy someday. With 64MB RAM and a Pentium 166mhz MMX CPU I think it will make an excellent server. Although I may upgrade the hard disk from 2.1GB to 20GB. I am undecided which distro I’ll install, probably Ubuntu 5.10 or Debian 3.1, but because the Libretto CT100 has weird hardware this install is not straight forward. The following web pages seem to be the most useful to me right now:

Installing Debian Linux 3.1 Sarge on Toshiba Libretto 50
Installing RedHat Linux on a Toshiba Libretto 100CT
- Has good background information on the hardware and hibernation partition

There are several other installs documented on the Linux on Toshiba Laptop and Notebook Computers page.

QEMU How-To Wiki Thingy

martin@wimpress.com (Martin Wimpress) — Mon, 05 Sep 2005 10:07:34 +0000

Adam Tricket got a few people interested in QEMU at the HantsLUG meeting on Saturday 3rd September 2005. Having recently discovered the wonders of QEMU myself I couldn’t help but get involved in the conversation. Me and my big mouth! I somehow managed to lumber myself with the job of writing a Wiki page about QEMU for the HantsLUG website. The page is more or less useful now and you can read it here…

LinuxHints/QemuEmulation

Herb Garden

martin@wimpress.com (Martin Wimpress) — Sun, 04 Sep 2005 10:04:52 +0000

I finally got around to planting a small herb garden today, something I have wanted to do ever since my wife and I designed the garden layout. The border looks much better for being planted out but, once they are established, I can’t wait to start harvesting my new crop 😄

So far I have planted Chive, Mint, Basil, Red Basil, Thyme, Lavender, Oregano, Sage, Rosemary and Parsley. I just need to add some Coriander.

Leaving Minerva

martin@wimpress.com (Martin Wimpress) — Thu, 01 Sep 2005 09:47:16 +0000

Yesterday was a sad day. I finally notified the core Minerva developers that I am leaving the project. Given that I started the project a few years back it was not a decision I made lightly.

I have a couple of reasons for leaving the project, the main one being that I just do not have the time available to effectively lead or contribute to the project. All open source projects require good leadership and I haven’t been able to provide that for most of 2005. It is with the best interests of Minerva in mind that I decided to leave.

However, Minerva is in very good hands and is under active development, with Dave and Chris leading the effort. I wish all the Minerva developers and users every success with the upcoming release of R4, it is a marvelous achievement and I am sure will prove to be very successful. I will be keeping an eye on R4 development, and if time permits, I may contribute some blocks and modules in the future. Until then, good luck and make me proud!

'ere ear!

martin@wimpress.com (Martin Wimpress) — Wed, 24 Aug 2005 09:46:37 +0000

I picked up a pair of new earbuds yesterday. I decided on the Sony MDR-EX71SL earbuds as all the reviews I have read have been very positive, and after listening to a few of my favourite albums with them, I am also very impressed with their performance. Due to their earbud design nearly all background noise is eliminated meaning you can hear your music very clearly at lower volumes, this good for preserving your hearing and also prevents you from annoying people nearby with noise bleed from the headphones. Bass performance is especially good and you get three different sizes silicon ear bud so everyone should be able to get a comfortable fit, the smallest size is best for me. I chose to get a black pair but the are also available in white for iPod owners.

Pocket Entertainment

martin@wimpress.com (Martin Wimpress) — Mon, 22 Aug 2005 09:36:08 +0000

I recently discovered ScummVM and have started playing Zak McKracken and the Alien Mindbenders for the first time in years! ScummVM allows you to run certain classic graphical point-and-click adventure games, provided you already have their data files, allowing you to play them on systems for which they were never designed!

So, I decided to organise my old adventure games for use with ScummVM so that I can enjoy them all again, but this has led to a couple of eBay bids to buy some games I am missing. I have used the ScummVM tools to compress as the game data files so I can fit the games on my iPAQ 4150. Life is good. I have been thinking of getting a new solid state Ogg Vorbis compatible portable music player but haven’t found one which completely suits my needs. So, my iPAQ 4150 is now standing in as my portable music player too. This is thanks too the excellent GSPlayer. I have configured my iPAQ 4150 to be a full time entertainment device.

I am off to get myself a decent set of head phones today, more on that later…

Introducing Stanley, a new theme for Jaws

martin@wimpress.com (Martin Wimpress) — Fri, 12 Aug 2005 09:29:53 +0000

I’ve finished hacking together a new theme for Jaws 0.5.x, which I have named Stanley, and is now the default theme on this site. I’ve still got to fix a couple of bugs and make sure that it is compatible with browsers other than Firefox. I wanted a theme which was crisp and clean, but also fully compatible with Jaws, so I decided to blend some of my favourite themes together. Stanley is inspired by the Kubrick theme which is popular among bloggers and although the Kubrick graphics have been borrowed the style sheet and layout are derived from the jaws, orange-grey and simple-green themes which are bundled with Jaws. As a result Stanley is properly Jaws compatible, in that left and right layouts work, all the CSS elements are supported and CSS tweaks are used throughout to improve the visual appeal.

Basic site layout complete

martin@wimpress.com (Martin Wimpress) — Mon, 08 Aug 2005 09:27:44 +0000

After fiddling around with the Jaws setup for a while I have finally settled on something which I’m happy with. Once I found the default_page setting in the registry I was very happy. I’ll enable more gadgets (plugins) as I find a use for them, but my next task will probably be creating my own theme or at least combining ideas from several existing themes.

Grrr, a site with teeth

martin@wimpress.com (Martin Wimpress) — Fri, 05 Aug 2005 09:10:10 +0000

Well, I have decided to get Flexion.Org on-line again and picked Jaws as the tool for the job. This might seem an odd choice considering I have been working on Minerva for the last 3 years, but there you are, life is strange.

Wimpy's World

faffing about with diffs

When AI promises go sideways

Enter faff, stage left

The plot twist that made me question everything

Why faff still matters (and it’s not just stubbornness)

Give it a whirl

Nothing but Nix

Introducing Nothing but Nix ️❄️

Space, the Final Frontier 🌌

The Slightly Mad Science Bit ‍🧑‍🔬

1. The Initial Slash: Instant Volume Creation

2. The Background Purge: Merciless Space Reclamation

BTRFS with nodiscard?

Choose Your Weapon: The Hatchet Protocol 🪓

From Frustration to Freedom ️🕊️

Getting Started with Nothing but Nix ✨

Beyond Storage to Virtual Machines? 🔮

Build Something Massive ️🏗️

Fuzzel Colour Picker

The Wayland Colour Picker Problem

Birth of fuzzel-hyprpicker

How It Works

Setting Up the Basics

The Clever Bit: SVG Icons

Menu Building

Putting It All Together

Integrating It With Your Setup

Possible Enhancements

The Joy of Keyboard-Driven Workflows

More Fuzzel-Based Tools?

Install ZeroTier on Steam Deck

Installing ZeroTier on Steam Deck

Enter the Desktop

Set a Password

Install Rwfus

Install ZeroTier

Enable and Start ZeroTier service

Join a ZeroTier Network

Use your ZeroTier network

Conclusion

Steam Box vs Steam Deck

As featured on Linux Matters! 🎙️

Hardware 🖥️

Node 202 Considerations 💼

GPU Considerations

CPU Considerations

Operating System 📀

ChimeraOS Experience 🦁

Steam Deck 🎮

Upgrades 📈

What’s next? 🔮

Conclusion 🏁

Flash Stadia Controller Bluetooth Firmware on Linux

Compatibility 🧩

Flash Stadia Controller Bluetooth Firmware using Ubuntu 22.04 🐧

Charge your controller

Unplug your controller to power it off

Hold the option button (three dots) while plugging in your controller

Press the Option, Assistant, A, and Y buttons at the same time

Select your controller in the Chrome device list and click “Connect”.

Download complete

Install Bluetooth mode

Installation complete

Pairing Stadia Controller with Bluetooth

Hurray up! ⏳️

Conclusion

Why I chose the ThinkPad Z13 Gen1 as my Linux laptop

As featured on Linux Matters! 🎙️

Past laptop purchasing mistakes 😱

Laptop criteria 📑

Lenovo ThinkPad Z13 Gen 1 with Ubuntu pre-installed

Specifications 📝

Build Quality & Design 💻️

Ubuntu pre-install experience 👌

Why not Ubuntu 22.04? 🤔

Linux Kernel 6.1 🌰

Battery endurance 🔋

What’s next? 🔮

Linux Matters

BTRFS with `nodiscard`?

Using `latest` version

Use `ENTRYPOINT` instead of `CMD`