Chaotic Experiments

Update: MSI Installer for GNU Emacs 29.3 for Windows

Fri, 29 Mar 2024 00:00:00 +0200

Following the GNU Emacs 29.3 release, my MSI installer for GNU Emacs has been updated!

This time there is no 32-bit version of it as no 32-bit executables were provided by the maintainers.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

I have also uploaded installers for 29.2 because I have completely missed that release.

Enjoy!

I decided to reuse screenshots from a previous release because the installer's code was not modified since then, so the screenshots will look the same (except the version number, obviously).

Update: MSI Installer for GNU Emacs 29.1 for Windows

Tue, 01 Aug 2023 00:00:00 +0300

Following the GNU Emacs 29.1 release, my MSI installer for GNU Emacs has been updated!

This time there is no 32-bit version of it as no 32-bit executables were provided by the maintainers.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

I decided to reuse screenshots from a previous release because the installer's code was not modified since then, so the screenshots will look the same (except the version number, obviously).

Update: MSI Installer for GNU Emacs 28.2 for Windows

Mon, 26 Sep 2022 00:00:00 +0300

Following the GNU Emacs 28.2 release, my MSI installer for GNU Emacs has been updated!

As usual, it is a belated update.

This time there is no 32-bit version of it as no 32-bit executables were provided by the maintainers.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

I decided to reuse screenshots from a previous release because the installer's code was not modified since then, so the screenshots will look the same (except the version number, obviously).

Update: MSI Installer for GNU Emacs 28.1 for Windows

Wed, 04 May 2022 00:00:00 +0300

Following the GNU Emacs 28.1 release, my MSI installer for GNU Emacs has been updated!

As usual, it is a belated update.

This time there is no 32-bit version of it as no 32-bit executables were provided by the maintainers.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

I decided to reuse screenshots from a previous release because the installer's code was not modified since then, so the screenshots will look the same (except the version number, obviously).

Update: MSI Installer for GNU Emacs 27.2 for Windows

Tue, 13 Apr 2021 00:00:00 +0300

Following the GNU Emacs 27.2 release, my MSI installer for GNU Emacs has been updated!

As usual, it is a belated update.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

I decided to reuse screenshots from a previous release because the installer's code was not modified since then, so the screenshots will look the same (except the version number, obviously).

Installing Entware on an Android-powered device: Entware Installer Module for Magisk

Fri, 02 Oct 2020 00:00:00 +0300

You may have an old unused Android-powered phone lying around, or you may have a set-top box which remains unused most of the time, so you want to make it to work also as a home WEB-server. Maybe you are geeky enough for an idea of carrying a Linux-powered server in your pocket. Or you would like to improve the functionality of your Android phone to include a package manager which simplifies installation of a large number of server-side software and development tools. These are only the reasons for installing Entware on an Android-powered device I can imagine. Everyone can have his or her reasons for doing so.

However, installing Entware on Android might be not straightforward. Initially, I planned to write a guide on how-to get Entware running on a device rooted using Magisk.

To my shame, the guide turned out to be complicated and error-prone. To save you from the hassle, I decided to reuse my findings and code and developed a module for Magisk which you can find on Github. Installing Entware using this module is as straightforward as possible, but for now, it supports only devices on which the root file system is a RAM-drive (rootfs). Unfortunately, this rules out most of the recent devices released with Android 9 and newer (so-called "system-as-root" devices). On these devices, the root file system is a real partition, so you need to manually modify it before using the module because Magisk does not currently provide a facility to make reversible modifications to directories outside of "/system."

I do not want to repeat myself in this post because I wrote hopefully complete documentation for the module. If you happen to have a newer device where the root file system is not a RAM-drive, I mentioned there the modifications which you might want to make before the installation.

Hopefully, I will get this module into the official Magisk modules repository at some point. Now you can install it manually, as described in the documentation.

Happy Entware installation!

Update to My 'daemonize' Library

Tue, 08 Sep 2020 00:00:00 +0300

I have updated my tiny C library for writing Unix daemons.

In fact, the library's code itself remained intact. When the original version was released, I made a somewhat sinful thing: although the library's code was written with portability in mind, the only example which came with it was Linux specific because of usage of signalfd(2) to redirect signal handling to the main event processing loop. This issue has been addressed recently: now, additionally to the original Linux specific example there is one which should be portable to the other Unix-like systems. It utilises the somewhat lesser-known self-pipe trick to redirect signal processing to the loop ¹.

I advise you to use the new example as a template for your daemons, even though there is slightly more code involved in doing the same thing. Even though we seem to live in the world of Linux monoculture, losing portability to other Unixes would be a sad thing.

Footnotes:

Here you can find another good example of the trick.

Update: MSI Installer for GNU Emacs 27.1 for Windows

Tue, 25 Aug 2020 00:00:00 +0300

Following the GNU Emacs 27.1 release, my MSI installer for GNU Emacs has been updated!

As usual, it is a belated update.

Although this release of Emacs contains a Windows installer, I decided to release my installer as well, because MSI installers might be more convenient in some cases.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

I decided to reuse screenshots from the previous release because the installer's code was not modified since then.

Finding the Square Root of a Number Without Using the Standard Library (Newton's Method)

Tue, 19 May 2020 00:00:00 +0300

In this short post, I want to show a way to find the square root of a positive number without relying on the standard library using Newton's method. This is the method which seems to be used in many implementations of the C standard library. The example code is in the C programming language. Before going to the implementation, I provide the mathematical formulae used.

I want to demonstrate how one could solve it with a relatively small amount of code without sacrificing precision much or at all compared to the sqrt() function from the C standard library.

A Bit of Math

Before going to the code, we need to describe the method which we are going to use. No advanced understanding of math is required.

Finding the square root of a number

\[N = \sqrt{X}\]

using Newton's method requires multiple steps.

To find the value of the desired accuracy

\[N = A_n\]

we will use the following formula:

\[A_{n} = \frac{A_{n-1} + \frac{X}{A_{n-1}}}{2}\]

where accuracy depends on the value of $n$. The higher the value of $n$ the more accurate value of square root we could obtain. We can say that $n$ is the number of approximations (how many times we want to apply this formula).

When choosing $n$ one needs to keep in mind that, in general, Newton's method has quadratic convergence which intuitively means that the number of correct digits roughly doubles in every step. On a computer, the number of approximations is dictated by the hardware.

From the formula above it is obvious that we need to know a way to find a value of $A_1$ to start.

A good starting value (or, should I say, a good first approximation) would be integer square root of $X$:

\[I=\lfloor \sqrt{X} \rfloor\]

\[A_{1} = I.\]

We will refer to this formulae during the programming process. Now let's go to the implementation.

Implementation

From the description above it becomes obvious that the first step which we need to do is to calculate the integer square root $I$. We can calculate this value iteratively increasing some number $k$ starting from $1$ until $k^2 \gt X$. Reaching this condition means that we have found the integer square root $I$ of the number $X$:

\[I = k - 1.\]

We can achieve this using the following C code:

unsigned long isqrt(const unsigned long X)
{
    unsigned long k = 1, result = 1;
    while (result <= X)
    {
        k++;
        result = k*k;
    }
    return k--;
}

There are more efficient methods available, but this one is easier to understand.

Then we need to find $A_n$¹ I have to admit that this sentence was added after some discussion at Reddit and it is quite bold. I tried to find some facts which will prove that I am wrong, but so far I have found none. But I am not a floating-point expert, though, so if you have information on this or, even better, the proof - please feel free to contact me. I am interested in improving this code and always ready to admit my mistakes. iteratively. Here the question arises: how should we peek $n$? We can paraphrase this question: how many times should we approximate the value?

In C we can use the following trick: considering that the floating-point numbers have finite precision, we may iterate until the value $A_n$ and $A_{n-1}$ will not become equal when compared using == operator in C. That means that we have reached the maximum possible precision on the current hardware (at least for this implementation). That is to say that I expect the method to converge as soon as

\[\left[A_{n-1} - A_n \right] \lt \epsilon\]

where $\epsilon$ is the machine epsilon for double-precision floating-point values (defined as DBL_EPSILON constant in C).¹

Please keep in mind that this approach may not work in some other programming environments. If it is the case, limit $n$ with some sane number. Also keep in mind, that we are targeting platforms with IEEE 754-compliant floating-point units.

So, finally, here is the code which implements calculation of the square root using Newton's method (with some checks omitted for clarity):

double nsqrt(double X)
{
    /* ... */
    unsigned long I;
    double A_prev, A;
    if (X == 0.0 || X == 1.0 /* .. */)
    {
        return X;
    }
    /* ... */
    I = isqrt((unsigned long)X); /* see above */
    if (X == (double)(I*I)) /* exact root */
    {
        return (double)I;
    }
    A = A_prev = I;
    do {
        A_prev = A;
        A = (A + X/A)/2.0;
    } while (A_prev != A);

    return A;
}

The reader might be interested in how well the code above performs compared against the sqrt() function from the C standard library. According to my tests, precision wise, it produces very similar or even the same results, at least on Windows (x86_64) and Linux (x86_64, ARM). I believe that it would be the case for Plan 9 as well. There is one notable exception though: when calculating $\sqrt{2}$ the results produced by the function above and the built-in one might differ. It seems that the functions in standard C library implementations may return the precomputed value of $\sqrt{2}$ (the M_SQRT2 constant), which might have been computed using different, possibly more precise, method.

Let's add the corresponding check to the code (as well as some other checks). Here is the revised version of the function:

double nsqrt(double X)
{
    const double inf = 1.0l/0.0l; /* positive infinity */
    const double ninf = -inf; /* negative infinity */
    const double nan = 0.0l/0.0l; /* not a number */
    unsigned long I;
    double A_prev, A;
    if (X == 0.0 || X == 1.0 || X == inf)
    {
        return X;
    }
    else if (X == 2.0)
    {
        return M_SQRT2;
    }
    else if (X == ninf || X < -0.0)
    {
        return nan;
    }
    /* NaN cannot be compared using == operator */
    else if (memcmp((void *)&X, (void *)&nan, sizeof(double)) == 0)
    {
        return nan;
    }
    /* I =  floor(sqrt(X)) */
    I = isqrt((unsigned long)X);
    if (X == (double)(I*I)) /* exact root */
    {
        return (double)I;
    }
    /* Newton's method: A{n} = (A{n-1} + X/A{n-1})/2 */
    A = A_prev = I; /* A{1} = I, or A{1} = floor(sqrt(X)) */
    do {
        A_prev = A;
        A = (A + X/A)/2.0;
    } while (A_prev != A);
    return A;
}

As far as I can tell, this version should be standard-compliant (at least on my hardware), but I do not want to make such a bold statement, especially considering that developing the standard compliant version of sqrt() was not the main point of this post.

Moreover, I do not want to claim that this function works in the same way as sqrt() from the standard C library. For some input values that can be not true. At very least you can expect that the results would be very similar. Please keep it in mind when comparing the implementations.

The main point of this post is to show that one can solve the problem without sacrificing precision with a small amount of code, a bit of math and some knowledge of how the hardware works.

The Full Example

/* to make assert() work in release builds */
#undef NDEBUG

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <assert.h>

#include <math.h>

/*
Given a number N, the task is to find the square root of N without using sqrt() function.
*/
static unsigned long isqrt(const unsigned long X)
{
    unsigned long k = 1, result = 1;
    while (result <= X)
    {
        k++;
        result = k*k;
    }
    return k--;
}

static double nsqrt(double X)
{
    const double inf = 1.0l/0.0l; /* positive infinity */
    const double ninf = -inf; /* negative infinity */
    const double nan = 0.0l/0.0l; /* not a number */
    unsigned long I;
    double A_prev, A;
    if (X == 0.0 || X == 1.0 || X == inf)
    {
        return X;
    }
    else if (X == 2.0)
    {
        return M_SQRT2;
    }
    else if (X == ninf || X < -0.0)
    {
        return nan;
    }
    /* NaN cannot be compared using == operator */
    else if (memcmp((void *)&X, (void *)&nan, sizeof(double)) == 0)
    {
        return nan;
    }
    /* I =  floor(sqrt(X)) */
    I = isqrt((unsigned long)X);
    if (X == (double)(I*I)) /* exact root */
    {
        return (double)I;
    }
    /* Newton's method: A{n} = (A{n-1} + X/A{n-1})/2 */
    A = A_prev = I; /* A{1} = I, or A{1} = floor(sqrt(X)) */
    do {
        A_prev = A;
        A = (A + X/A)/2.0;
    } while (A_prev != A);
    return A;
}

int main(int argc, char **argv)
{
    const double inf = 1.0l/0.0l; /* positive infinity */
    const double ninf = -inf; /* negative infinity */
    const double nan = 0.0l/0.0l; /* not a number */
    size_t i;
    /* some edge cases */
    assert(isnan(sqrt(nan)));
    assert(isnan(nsqrt(nan)));
    assert(isnan(sqrt(-1.0)));
    assert(isnan(nsqrt(-1.0)));
    assert(sqrt(inf) == nsqrt(inf));
    assert(isnan(sqrt(ninf)));
    assert(isnan(nsqrt(ninf)));
    assert(sqrt(0.0) == nsqrt(0.0));
    assert(sqrt(-0.0) == nsqrt(-0.0));
    assert(sqrt(1.0) == nsqrt(1.0));
    assert(sqrt(2.0) == nsqrt(2.0));

    for (i = 0; i < 100; i++)
    {
        const double r = sqrt(i), nr = nsqrt(i);
        printf("%lu - sqrt(): %.50f, nsqrt(): %.50f. same: %s\n",
                i, r, nr, (r == nr ? "true" : "false"));
    }
    return 0;
}

Footnotes:

I have to admit that this sentence was added after some discussion at Reddit and it is quite bold. I tried to find some facts which will prove that I am wrong, but so far I have found none. But I am not a floating-point expert, though, so if you have information on this or, even better, the proof - please feel free to contact me. I am interested in improving this code and always ready to admit my mistakes.

Facilities for Context Switching and Coroutines on AVR microcontrollers (and AVR-based Arduino boards)

Fri, 15 May 2020 00:00:00 +0300

In this short post, I want to present a small project on which I have been working for the last couple of weeks. It is a small library to control an AVR micro-controller execution context. Additionally, it can be used on AVR-based Arduino boards.

I have to note that this is my first serious attempt in micro-controllers programming. Nevertheless, I tried to do my best, but bugs are always possible, so bug reports are welcome. Caveat lector.

Initially, the project started as a side effect of my attempt to understand how task switching is performed in operating systems. I believe that it is better to use microcontrollers to understand the concepts like this one because the generic computers are meant to run full-featured, multi-user operating systems with virtual memory support. These things, while important on their own, considerably complicate the low-level programming to the point where it is hard to see the forest behind the trees.

The functionality, developed in the process, turned out to be generic enough, so I decided to continue to work on the project, and implemented substitutes for functions getcontext(), setcontext(), makecontext(), swapcontext(). These functions are commonly available on UNIX-like operating systems because they used to be a part of the POSIX standard (You can read about them here and here). They can be used to implement many interesting abstractions, but in most cases, this functionality is used to implement coroutines. I decided to go ahead and implemented the asymmetric stackful coroutines (as in the Lua programming language).

I tried to make the library's interface as easy to use as possible. There are 8 functions (4 to work with execution context, 4 for to work with coroutines), 4 macros and 2 data types. The interface can be summarised as follows (to give you the high level view):

/* Context Switching: avrcontext.h */

typedef struct avr_context_t_ {
    /* ... */
} avr_context_t;

typedef void (*avr_context_func_t)(void *);

#define AVR_SAVE_CONTEXT(presave_code, load_address_to_Z_code) /* ... */
#define AVR_RESTORE_CONTEXT(load_address_to_Z_code) /* ... */

#define AVR_SAVE_CONTEXT_GLOBAL_POINTER(presave_code, global_context_pointer) /* ... */
#define AVR_RESTORE_CONTEXT_GLOBAL_POINTER(global_context_pointer) /* ... */

void avr_getcontext(avr_context_t *cp);
void avr_setcontext(const avr_context_t *cp);

void avr_swapcontext(avr_context_t *oucp, const avr_context_t *cp);
void avr_makecontext(avr_context_t *cp,
                     void *stackp, const size_t stack_size,
                     const avr_context_t *successor_cp,
                     avr_context_func_t funcp, void *funcargp);

/* Asymmetric Stackful Coroutines: avrcoro.h */

typedef struct avr_coro_t_ {
    /* ... */
} avr_coro_t;

typedef enum avr_coro_state_t_ {
    AVR_CORO_SUSPENDED = 0,
    AVR_CORO_RUNNING,
    AVR_CORO_DEAD,
    AVR_CORO_ILLEGAL,
} avr_coro_state_t;

typedef void *(*avr_coro_func_t)(avr_coro_t *, void *);

int avr_coro_init(avr_coro_t *coro,
                  void *stackp, const size_t stack_size,
                  avr_coro_func_t func);
int avr_coro_resume(avr_coro_t *coro, void **data);
int avr_coro_yield(avr_coro_t *self, void **data);
avr_coro_state_t avr_coro_state(const avr_coro_t *coro);

I do not want to repeat myself here, so if you are interested, you can find the full project documentation on the project's page on GitHub. There are examples included. I tried to arrange them in such a way that one can use them as a tutorial. They are, I hope, properly documented as well.

The list of available examples:

Context Switching:
Coroutines:

The library can be installed via "Library Manager" in Arduino IDE (search for AVR-context).

The project turned out to be very interesting on itself and involved some research. The references to the specifications, papers and other resources can be found in the project's documentation. I may do a couple of publications which demonstrate how this library was developed. Although the result is hardware specific, the approach itself can be adapted to almost any hardware platform.

High-Quality Video Playback on Linux

Mon, 07 Oct 2019 00:00:00 +0300

Some programs are so damn good that you start to see them without any relation to the platform on which they run. In this case, you start looking for their substitutes if you are unable to use them on some platforms, often without success. In my opinion, one of the better combinations for video playback is Media Player Classic Home Cinema in combination with MadVR video renderer. It is rather hard to beat this combination, but there is a very good, open-source alternative available on Linux: mpv+SMPlayer. In this post, I am going to describe how I essentially repeated my configuration for high-quality video playback on Linux.

I believe that there is one thing worth noting: as both SMPlayer and mpv are available on Windows the information from this post might be useful if you plan to use these programs on Windows as well. I think that no additional adaptation required for this platform. The same goes for macOS.

I would suggest you keep the man page of mpv opened so that you can quickly read about the configuration options referenced in this post.

Important Options for High-Quality Video Playback

To improve video playback quality, we need the following:

Good scaling algorithms - I believe that this part is obvious. I should note that scaling algorithms are computationally intensive in general and require a fairly high-performance video card. I suggest that you use the following algorithms: ewa_lanczos (aka jinc) with an anti-ringing filter for up-scaling and chroma scaling, mitchell for down-scaling. A better alternative to ewa_lanczos would be ewa_lanczossharp, but it is very computationally intensive. If you have decided to use it, I believe that you might not need the anti-ringing filter in that case.
Smooth-motion (with V-Sync). Here I should note that this is not converting video to the higher frame-rate as some might think. This technique allows frames from a video to better synchronise with frame-rate of your display. Without this, some judder might appear during motion. This technique is called Interpolation in mpv's manual, and there is an in-depth explanation of it on the wiki of the project.
Debanding filter - prevents colour banding from appearing on the scenes containing some gradients.

Configuring mpv

Let's create a configuration file for mpv which uses the techniques mentioned above. The configuration file of mpv is located here: ~/.config/mpv/mpv.conf, we are going to add the options described below into it.

In some cases, I am going to mention settings both for middle-end and high-end systems. In my opinion, the visual difference between them is so subtle that I would suggest that you go with the configuration for middle-end systems first, especially if you use a laptop. If you use a relatively modern desktop or do not mind extra battery drain and fan noise then go experiment with the configuration for high-end systems.

The first thing which we should set is a video renderer profile. There are many of them but gpu and gpu-hq are the ones in which we are interested the most. These, as you might have guessed, allow offloading most of the video post-processing and rendering to a GPU using OpenGL or Vulkan API. gpu-hq is pre-configured to use higher quality, more computationally intensive filters.

middle-end systems:

profile=gpu

high-end systems:

profile=gpu-hq

We may use Vulkan or OpenGL as a GPU API. Honestly, I have not noticed any difference in performance when playing a video using either, so I would suggest that you stick with OpenGL as the most compatible option. We are going to enable V-Sync as well.

gpu-api=opengl
opengl-swapinterval=1

If you want to use Vulkan instead, here is how to do it:

gpu-api=vulkan
vulkan-swap-mode=fifo

The next logical step is to configure smooth-motion. Here is how:

video-sync=display-resample
interpolation
tscale=oversample

Now let's configure scaling.

middle-end systems:

scale=ewa_lanczos:scale-antiring=1
cscale=ewa_lanczos:scale-antiring=1
dscale=mitchell
correct-downscaling

high-end systems:

scale=ewa_lanczossharp
cscale=ewa_lanczossharp
dscale=mitchell
correct-downscaling

It is a good idea to enable dithering when it is necessary. mpv uses a fairly good dithering algorithm by default.

dither-depth=auto

The last step is to offload video decoding to GPU when possible. It is especially useful for laptops. We are using auto-copy instead of auto to enforce copying back decoding results back to the system RAM. It might be necessary for some filters to work and helps to avoid some decoding issues at the expense of a relatively small performance loss.

hwdec=auto-copy

Using SMPlayer with a Pre-configured mpv Instance

I prefer to use mpv as a back-end of SMPlayer. If you are going to use mpv as a standalone video player then you can skip reading this section.

Modern versions of SMPlayer can use mpv as its back-end without any additional configuration. However, SMPlayer tries to have mpv under its full control and starts it with --no-config option. It forces mpv to ignore its default configuration file. Fortunately, SMPlayer can pass user-defined command-line options to mpv, which allows us to bypass this limitation. To do it we have to use this extra option to force mpv to load its configuration file:

--include=~/.config/mpv/mpv.conf

Here is the screenshot which shows us how to do it:

Please keep in mind that SMPlayer expects that mpv does not load its configuration file, so be conscious about what you have in the mpv.conf file.

Also, it might be a good idea to configure SMPlayer to use Default video output engine with disabled post-processing:

Conclusions

In this post, we have configured mpv and SMPlayer for high-quality video playback. I think that there are some things left which are worth noting.

Firstly, the video output quality depends mostly on the quality of the video files. No amount of configuration will make a miracle here if all you have is a low-quality media material.

Secondly, the quality of the display matters a lot. It does not necessarily mean that you should use these ridiculously expensive professional displays for high precision colour work. Adequately priced display with good IPS matrix will do. I doubt that you will see any difference on a low-end TFT monitor.

I think that these things are rather obvious, but I wanted to have them mentioned anyway.

Hopefully, this post will improve your ~~anime~~ video playback experience and make your entertainment more joyful.

Update: MSI Installer for GNU Emacs 26.3 for Windows with Per-User Installation Support

Tue, 01 Oct 2019 00:00:00 +0300

I have been working on improving Emacs installer for Windows to add support both per-user and per-machine installations. It was made to address the installation issues of users without administrative privileges. This post is a follow up to a post I have made recently.

This effort resulted in adding support for building per-user installers. These are available alongside the usual, per-machine installers. It should be noted, though, that their name might be a little bit deceptive because they support installation in both per-machine and per-user mode. As you might expect the latter does not require administrative privileges.

Feature-wise both kinds of installers provide similar functionality with one notable exception: the per-user installers unable to set and modify environmental variables conditionally depending on selected scope and, thus, unable to add installation directory of Emacs to the Path. This is a limitation of tooling.

Please keep in mind that mixing both per-user and per-machine installations of the same package on the same machine might result in strange behaviour. I would suggest sticking with classical, per-machine only installers if you have enough privileges on your machine.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

As usual, If you used an older version of the installer you should be able to perform an in-place upgrade.

The screenshots provided below show the installation of a per-user installer in advanced mode.

Enjoy!

Update: MSI Installer for GNU Emacs 26.3 for Windows

Thu, 12 Sep 2019 00:00:00 +0300

Following the GNU Emacs 26.3 release, my MSI installer for GNU Emacs has been updated!

This time it is rather a belated update.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

Integrating Xfce with LightDM Display Manager

Fri, 19 Jul 2019 00:00:00 +0300

One of the most basic security features of any laptop or workstation is the ability to lock the sscreen. It might be convenient in many other, non-security related, scenarios as well. In most cases, this feature works without user's intervention on most of the Linux distributions, yet it did not work for me in Xfce4 with LightDM on Arch Linux. As I understand, Xfce does not have its display manager, so it relies on third-party solutions, mostly Gnome Display Manager (GDM). As I prefer Xfce as my desktop environment, I did not want to install GDM as it needs many Gnome libraries to work. Because of this, I decided to use LighDM as my display manager, which is more desktop environment agnostic and lightweight. Unfortunately, Xfce lacks proper integration with it (at least in Arch Linux). In this post, I want to provide a solution to this problem. Moreover, I believe that information in this post might give some ideas to the users of the other desktop environments and window managers on how to integrate them with LightDM.

To some extent, this post is a continuation of the other LightDM related post which I have posted earlier.

If the first two buttons from the image below do not work on your system, you might want to continue reading this post.

How Xfce Initiates Screen Locking and User Switching

Xfce does not do screen locking or user switching itself, it delegates this functionality to the other programs.

For screen locking Xfce executes xflock4 program which is a shell script. The script locks the session by calling the following programs if they are present on the system (in the following order):

xscreensaver-command
gnome-screensaver-command
xlock
slock

Similarly, Xfce delegates user switching to the gdmplexiserver which is (or was) a part of the Gnome Display Manager package. As you can see, Xfce is attuned to reuse some of the parts of the GNOME desktop environment.

Considering that LightDM has its utility to control itself, the dm-tool, we might try to "deceive" Xfce to invoke the dm-tool instead. We could accomplish that by placing the scripts which call the dm-tool with a required parameter (lock or switch-to-greeter) and are named as above mentioned programs into a directory from the $PATH environmental variable (/usr/local/bin seems like the right choice).

The Scripts to Perform Screen Locking and User Switching

At this point, it is obvious that we need two executable shell scripts, one for session locking and one for user switching, which we should place into /usr/local/bin folder. For the first one, I propose to use the name slock or xlock (I used to recommend xscreensaver-command, but I had some problems in SMPlayer in this case). Here is the code:

#!/bin/sh

# fake slock to make xflock to use lightdm locker

DMTOOL_AVAILABLE="$(which dm-tool)"

if [ -n $DMTOOL_AVAILABLE ] ; then
    dm-tool lock
fi

For the second case, user switching, we might use a very similar script, named as gdmplexiserver (the only difference is a parameter to the dm-tool):

#!/bin/sh

# fake gdmplexiserver to make xflock to use lightdm locker
# for user switching

DMTOOL_AVAILABLE="$(which dm-tool)"

if [ -n $DMTOOL_AVAILABLE ] ; then
    dm-tool switch-to-greeter
fi

If you are a systemd user, you might try to use a script which relies on loginctl for both cases:

#!/bin/sh

LOGINCTL_AVAILABLE="$(which loginctl)"

if [ -n $LOGINCTL_AVAILABLE ] ; then
    loginctl lock-session
fi

After you have done all the steps mentioned above, screen locking and user switching in Xfce should work as expected.

UPD:

20.07.19 - Note about problems with xscreensaver-command.

Using LightDM with an External Monitor on a Laptop or with the Multi-monitor Setups

Tue, 16 Jul 2019 00:00:00 +0300

Like many other users, I find it convenient to connect an external monitor to my laptop sometimes. As I often do this, I expect this functionality to work flawlessly but, unfortunately, I experienced some problems when using LightDM on Arch Linux . By default, LightDM tries to mirror data on multiple displays, and it does not work well when you have monitors with different resolutions. In this post, I want to share my solution to this problem. I expect that this solution might be useful not only for laptop users but for the workstation users with multi-monitor setups as well. Moreover, if LightDM's user interface sometimes stops responding to your actions, this post might be useful for you.

In general, on a laptop with an external monitor, I expect the login screen to appear on it and have the laptop screen disabled in that case. If there are multiple external monitors connected to the system, I want only one of them used for a login screen.

Fortunately, these actions can be programmed using a UNIX shell script build around the functionality of the xrandr utility.

When used without any parameters it prints the list of monitor outputs alongside the information about the connected monitors:

artem@artbox:~$ xrandr
Screen 0: minimum 8 x 8, current 1366 x 768, maximum 32767 x 32767
LVDS1 connected 1366x768+0+0 (normal left inverted right x axis y axis) 340mm x 190mm
   1366x768      60.03*+
   1280x720      59.74
   1024x768      60.00
   1024x576      60.00    59.90    59.82
   960x540       60.00    59.63    59.82
   800x600       60.32    56.25
   864x486       60.00    59.92    59.57
   640x480       59.94
   720x405       59.51    60.00    58.99
   680x384       60.00
   640x360       59.84    59.32    60.00
DP1 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
VGA1 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)

We could filter out the information about the available outputs using the UNIX tools:

artem@artbox:~$ xrandr | grep connected | cut -d ' ' -f 1-2
LVDS1 connected
DP1 disconnected
HDMI1 disconnected
VGA1 disconnected
VIRTUAL1 disconnected

Keeping in mind that an internal laptop screen is usually connected using the LVDS interface, we could use the information obtained from the xrandr tool to achieve what is required.

Let's outline the logic of the script using pseudo-code:

var primary_out = nil
var to_disable = nil

foreach monitor_out in list_monitor_outputs() do
    if is_connected(monitor_out) then
        if is_connected_via_LVDS(monitor_out) then
            to_disable = monitor_out
        else if primary_out == nil then
            primary_out = monitor_out
        else
            disable_monitor_output(monitor_out)
        endif
    endif
endfor

if primary_out != nil and to_disable != nil then
    set_as_primary_monitor_output(primary_out)
    disable_monitor_output(to_disable)
else if primary_out != nil then
    set_as_primary_monitor_output(primary_out)
endif

After translating the pseudo-code above into a shell script, I got the following code:

#!/bin/sh

# the script will not work if xrandr is not available
if [ -z "$(which xrandr)" ]; then
    exit 1
fi

DISABLE_SUBSTRING="LVDS"

TO_DISABLE=""
PRIMARY_OUT=""

old_IFS="${IFS}"
IFS=$'\n'
for out in `xrandr | grep connected | cut -d ' ' -f 1-2`; do
    name=`echo $out | cut -d ' ' -f 1`
    state=`echo $out | cut -d ' ' -f 2`
    if [ "$state" = "connected" ]; then
        case "$name" in
            # disable a monitor connected via LVDS
            ${DISABLE_SUBSTRING}*)
                if [ -z "$TO_DISABLE" ]; then
                    TO_DISABLE="$name"
                fi
                ;;
            *)
                # set the 'first' external monitor as the
                # primary one, disable the other external monitors
                if [ -z "$PRIMARY_OUT" ]; then
                    PRIMARY_OUT="$name"
                else
                    xrandr --output "$name" --off
                fi
                ;;
        esac
    fi
done
IFS="${old_IFS}"

# If there is an internal screen, as well as at least one external monitor, then
# disable the internal monitor and use the external monitor as the primary one.
#
# Otherwise, if there are any external monitors, use the 'first' detected one
# as the primary monitor.
if [ -n "$TO_DISABLE" ] && [ -n "$PRIMARY_OUT" ]; then
    xrandr --output "$PRIMARY_OUT" --primary --auto --output "$TO_DISABLE" --off
elif [ -n "$PRIMARY_OUT" ]; then
    xrandr --output "$PRIMARY_OUT" --primary --auto
fi

exit 0

To use the script, one should save it into an executable file and alter the LightDM's settings to use it to configure the displays.

In order to accomplish this one should change the display-setup-script in the [Seat:*] section of the LightDM configuration file (/etc/lightdm/lightdm.conf):

[Seat:*]
...
display-setup-script=/path/to/the-script
...

To apply the new settings, please restart the LightDM daemon. The next LightDM session should use the script to configure monitors.

Update: MSI Installer for GNU Emacs 26.2 for Windows

Thu, 18 Apr 2019 00:00:00 +0300

Following the GNU Emacs 26.2 release, my MSI installer for GNU Emacs has been updated!

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

If you used an older version of the installer you should be able to perform an in-place upgrade.

Enjoy!

Querying Windows Registry from Emacs Lisp

Fri, 05 Apr 2019 00:00:00 +0300

Recently I needed to query registry values from within Emacs. Emacs, being a cross-platform application mostly tailored to work on UNIX-like operating systems, lacks this functionality. As you can expect, I ended up writing a code which accomplishes the task.

One might need this functionality to configure Emacs to automatically find installation paths of the other tools. For example, I use it to get the git installation path to configure Magit properly.

Although I needed to query only string values (REG_SZ), there is support for the other registry value types as well. The provided code uses standard reg.exe command line utility to read values from the registry.

(defun win-reg-read (regkey &optional value access-64-bit)
  "Read a value in the Windows registry. If the value does not exist, it returns nil. For an empty REG_BINARY value it returns T."
  (when (string-equal system-type "windows-nt")
    (let* ((regkey (encode-coding-string regkey locale-coding-system))
           (value (and value (encode-coding-string value locale-coding-system)))
           (reg-exe (concat (getenv "WinDir") "\\System32\\reg.exe"))
           (bits-64-param (if access-64-bit
                              "/reg:64"
                            "/reg:32"))
           (reg-value (let ((output (with-output-to-string
                                      (with-current-buffer
                                          standard-output
                                        (if value
                                            (process-file reg-exe nil t nil
                                                          "query"
                                                          regkey
                                                          "/v"
                                                          value
                                                          bits-64-param)
                                          (process-file reg-exe nil t nil
                                                        "query"
                                                        regkey
                                                        "/ve"
                                                        bits-64-param))))))
                        (set-text-properties 0 (length output) nil output)
                        output)))
      (let* ((lines (split-string reg-value "\n" nil nil))
             (result-line (nth 2 lines)))
        (when (> (length lines) 4)
          (let* ((result-line (if value ; skip value name
                                  (cl-subseq result-line (+ 8 (length value)))
                                (replace-regexp-in-string "^[ ]+[\(][^\)]+[\)][ ]+" ; default value
                                                          ""
                                                          result-line)))
                 (tokens (split-string result-line "    " t))
                 (type-string (car tokens)) ; get registry value type
                 (result (cl-subseq result-line (+ 4 (length type-string))))) ; get registry value
            ;; decode the registry value string into a lisp object
            (cond
             ((or (string-equal type-string "REG_DWORD")
                  (string-equal type-string "REG_QWORD"))
              (string-to-number (cl-subseq result 2) 16))
             ((or (string-equal type-string "REG_SZ")
                  (string-equal type-string "REG_EXPAND_SZ"))
              result)
             ((string-equal type-string "REG_MULTI_SZ")
              (split-string result (regexp-quote "\\0") t)) ; will not work if there are '\0' sequences in the string
             ((string-equal type-string "REG_BINARY")
              (let ((res)
                    (pos 0)
                    (size (/ (length result) 2)))
                (if (zerop size)
                    t ; value is present, but empty
                  (progn
                    (dotimes (i size)
                      (push (string-to-number (cl-subseq result pos (+ pos 2)) 16) res)
                      (incf pos 2))
                    (reverse res)))))
             (t nil))))))))

The function above accepts three arguments:

Registry key name;
Registry value name (optional). If omitted, the default value of the registry key is queried;
The last parameter helps to handle registry virtualisation on 64-bit versions of Windows. If true, it queries the 64-bit branch of the registry, 32-bit one otherwise.

The code returns the value on success, NIL otherwise. For values, which by their nature hold multiple data items (REG_MULTISTRING_SZ, REG_BINARY) the lists of values are returned.

N.B. Eli Zaretskii pointed out that there were some problems with handling text encodings in the older version of the code. Hopefully, they are fixed now.

Limitations

I should make a couple of notes about accessing the multi-string (REG_MULTISTRING_SZ) values and binary values (REG_BINARY). The reg.exe utility reports information for multi-string values in such a way that the strings are separated by the "\0" sequence. If there is this sequence somewhere in the data the code below will return incorrect results.

Empty binary registry values are reported in a somewhat weird way: the code returns T for them.

I doubt that these limitations would bite you in practice.

Dealing with Registry Virtualisation

On 64-bit versions of Windows, the access to some registry keys is virtualised for 32-bit applications. As many applications exist in both 32-bit and 64-bit variants often it makes sense to check for both values in the registry.

The code below does just that. On 64-bit versions of Windows, it tries to query value in the 64-bit section of the registry at first and, if it is absent, it tries to query the 32-bit one. I guess that in most cases it is what you want.

(defun win-reg-read-64-or-32 (regkey &optional value)
  "Read a value in the Windows registry.
   If the value does not exist, it returns nil.
   Checks both 32-bit and 64-nit keys,
   considers system bitness when searching for a value."
  (let ((check (getenv "ProgramW6432")))
    (if (or (null check) (zerop (length check)))
        (win-reg-read regkey value nil) ; 32-bit Windows
      (or (win-reg-read regkey value t) ; 64-bit Windows
          (win-reg-read regkey value nil)))))

A Couple of Examples

You might try to get the Git For Windows installation path this way:

;; Installation Path
(win-reg-read-64-or-32 "HKLM\\SOFTWARE\\GitForWindows" "InstallPath")
"C:\\Tools\\Git"
;; Git core. Magit Expects it to be in the EXEC-PATH
(win-reg-read-64-or-32 "HKLM\\SOFTWARE\\GitForWindows" "LibexecPath")
"C:\\Tools\\Git\\mingw64\\libexec\\git-core"

The same thing for LibreOffice (useful if you want to configure exporting org-mode files to ODT):

(win-reg-read-64-or-32 "HKEY_LOCAL_MACHINE\\SOFTWARE\\LibreOffice\\UNO\\InstallPath")
"C:\\Program Files\\LibreOffice\\program"

You could obtain the Arduino Integrated Development Environment installation path in a similar way:

(win-reg-read-64-or-32 "HKEY_LOCAL_MACHINE\\SOFTWARE\\Arduino" "Install_Dir")
"C:\\Tools\\Arduino"

I hope you got the idea. Not every application keeps its installation path in the registry though, but many do. In any case you should do your research using the Windows Registry Editor.

Off course you could use this code not only for the purpose of querying of the installation paths. I believe that the ability to read registry values from within Emacs makes it possible to configure it in such a way that it would barely need any tailoring the configuration for the specific machine.

I hope you will find the code provided above to be useful or, at least, interesting.

P.S. Future versions of Emacs will directly support reading from registry. There are a couple of limitations though. Thanks to skeeto for pointing out.

Fixing Sleep Mode Issues in Windows 10 (8.1/8) by Disabling InstantGo Mode

Thu, 14 Mar 2019 00:00:00 +0200

Recently I have encountered a problem while using a new laptop - sleep mode did not work reliably. Sleep mode is an important feature for me, so this state of affairs was not acceptable. The fix was rather illogical and it is impossible to accomplish via settings in the graphical user interface tools provided by the Windows operating systems. If you have newer hardware and ever encountered the issues when your Windows-based laptop will not go into sleep mode or will not wake up reliably, or you would put your laptop into sleep mode only to find two hours later that it is active and hot in your backpack - please, continue reading. Probably I have a solution for you worth investigating.

If you haven't encountered any problems with sleep mode on your hardware - this article is not for you. Otherwise, feel free to continue, but remember that as Windows-powered hardware is diverse and it is impossible to provide a fix which would work reliably for anyone. Moreover, this kind of problems might be a result of failing hardware or buggy drivers. Please, keep it in mind and proceed at your own risk! Caveat Lector.

Background Information

After some research, I found a solution. On my laptop the problem turned out to be in InstantGo mode (formerly named as Connected Standby). This mode is available on newer hardware and allows to put it in a less power consuming mode while disabling some of the internal devices in a way very similar to how smart-phones and tablets do. It should be noted that network connections remain active so a user still would not miss any notifications in programs like Skype, Facebook Messenger, Steam, mail client, etc. This is why this mode used to be named as Connected Standby.

This mode might look great on paper, but the fact is that laptop hardware significantly differs from the hardware of hand-held devices. A quick search revealed that I am not the only one who have experienced similar problems. There were users, who reported similar problems even on the hardware which was designed by Microsoft (aka Sleep of Death). There is no doubt that such a mode might be useful for someone, but I cannot understand two things about it:

Why it is enabled by default?
Why there is no obvious way to configure it?

I think that here, by trying to enforce this mode by default instead of "classical" sleep mode, Microsoft made a poor decision. When you sleep, you are not expected to react to any external events, right? So should your laptop, otherwise, this mode cannot be named as "sleep mode" any more as it makes no sense.

So, if InstantGo mode is not reliable for you, you might want to try to disable it. Doing so will bring the "classical" sleep mode back. At least on my hardware, it works reliably and it never failed. The fact that there is no method to easily disable InstantGo mode further complicates the matter.

The Possible Fix

If you are tech-savvy please go ahead and read this article (by the way, I have registry files to disable and enable InstantGo). If you are not, please continue reading because I have detailed instruction for you:

Open the Start Menu (press the Win button on your keyboard) and start typing regedit.exe.

Find the regedit.exe in the search results, click on it with the right mouse button, choose "Run as Administrator".

A confirmation window might appear (it depends on your system's configuration). It might be required to enter the administrator's account password here. Enter it, if it is required, and answer affirmatively.

The "Registry Editor" window should appear.

Navigate to the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power.

Find the CsEnabled registry value. Please remember to what it is set (usually it is set to 1).

Double-click on the CsEnabled registry value. Small windows with the title "Edit DWORD (32-bit) value" should appear. Set the value to 0 and click "OK".

Reboot the computer.

Hopefully, after performing the steps written above, sleep mode on your laptop should be more reliable than it was before. If it isn't, please repeat the steps but set the CsEnabled back to its original value - probably InstantGo was not the cause of your problems with the sleep mode.

Conclusion

In general, I orient my blog towards programmers, IT personnel and, to a lesser extent, tech-savvy users. Arguably, tracking down this problem was difficult and I realised that probably there are less advanced users who might encounter the same issue but lack the necessary knowledge and research skills to figure out what is going on. This article is my humble attempt to improve the situation.

I believe that it is a major issue because under some circumstances it might lead to damaged hardware (e.g. when you often carry your laptop in a bag or a backpack).

I think that Microsoft targeted the InstantGo (Connected Standby) mostly towards less tech-savvy users who expect their computer to perform some tasks even when it is in the sleep mode (as smart-phones and tablets do), but, I believe, it is safe to say that in this particular case the cure was worse than the illness.

MSI Installer for GNU Emacs

Sat, 05 Jan 2019 00:00:00 +0200

If you use GNU Emacs on Windows you might have noticed that on the official site the only download option for this platform is a ZIP-archive, no installer is provided. It is, of course, a very compatible option and, probably, most of the GNU Emacs users can integrate it with the operating system manually. Nevertheless, I think that there is no need to make it harder than it should be. It would be great if there was an installer.

Considering that I managed to built an installer using open source WiX Toolset for the other project which I help to maintain (Corman Lisp), I decided to build an MSI installer for GNU Emacs as well. It was rather easy to adapt existing code for the new project.

This project is my humble contribution to the GNU Emacs community.

You can find all the sources on GitHub as well as the pre-built installers for the latest release of GNU Emacs. The sources are in the public domain (CC0). The repository also includes the build instructions. The provided installers bundle official, unmodified GNU Emacs binaries for Windows.

The availability of an MSI installer for an application makes it easier to deploy in some environments.

This project addresses not only the GNU Emacs's installation experience. It might be interesting to the developers which are facing the task of building an MSI installer. It seems that it is not easy to stumble upon a complete Windows Installer XML project. If it is the case, you might want to have a look at the Corman Lisp installer project which is very similar yet somewhat more complex.

Please note that I am not a Windows Installer guru, but I do eat my own dog food, so this installer was tested at least by one person before the release.

Enjoy!

Personal Notes on Corman Lisp 3.1 Release

Sun, 30 Dec 2018 00:00:00 +0200

Today, as I announced earlier, the Corman Lisp 3.1 has been released after more than a decade-long break. Somehow it happened that I am one of the active maintainers of the Corman Lisp, so, I think, it might be appropriate to make a post on this site about how I became a maintainer, what motivated me to work on it and why I believe it is important. You will not find the release notes here because I think it is not the right place to post them. If you have at least mild interest in this implementation, I would suggest you to continue reading.

How I became a maintainer

Some time ago I stumbled upon a non-supported Common Lisp implementation on Github, Corman Lisp. It was in a bad shape - it did not work well on the modern, 64-bit versions of Windows and the project's solution needed a bit of cleaning. I should note that I am genuinely interested in Lisp, compilers, programming languages and their runtime environments. Additionally to that, I possess some knowledge of Windows application programming interface, a skill which not every system programmer, interested in open source development, seems to have. The project seemed to fit my interests well, so I decided to look at it more closely.

Three days later I fixed a very annoying bug which made it possible to use the software on the 64-bit versions of Windows and things started to look even more interesting: it turned out I can work on such a complex system and make important changes to it. I should note that this was kind of an act of insight, to this very day I do not understand how I managed to fix it. Later Roger Corman, the original author, contacted me via Github, thanked for fixing this and returned to the project.

Additionally to that, I understood that this project is very important from the educational perspective. This is a fairly complete Common Lisp implementation which is easy to study and might give one interesting inside on how to build a working, high-performance environment for a dynamic programming language which includes, among other things, a native code compiler and a garbage collector. I realised that if I am able to understand how it works, then someone else can do that too. For me, it was a major turning point. Then I contacted Zachary P. Beane (a.k.a. Xach) who gave me the write permissions to the repository and I became a co-maintainer of the Corman Lisp.

Later Luis Cervantes joined the project and made a lot of valuable contributions.

Why I think that maintaining Corman Lisp is important

There are multiple reasons why I believe that maintaining Corman Lisp is important. I am going to describe the main ones in more detail.

It is a good implementation for newbies

In my opinion, Common Lisp is not a hard programming language to learn, especially comparing it to such a language as C++. What is hard is getting the proper environment at the beginning. Here a potential newbie typically has the two general recommendations from the community.

The first one is to use Emacs+SLIME+SBCL (or Clozure CL, …). I should say that this is what I, and most of the other Common Lisp programmers, use most of the time and for a reason: it is an incredibly powerful development environment, probably the best one among environments for the dynamic programming languages. On the other hand, it might be difficult to set up and Emacs might seem weird for a beginner. It takes some time to tame it. In fact, to manually configure Emacs as a Common Lisp development environment one should be familiar with another Lisp (Emacs Lisp).

To overcome this issues a newbie must not lose the spark of interest from the very beginning. For someone, this experience might be overwhelmingly frustrating. It should be noted that such a user might be potentially a good programmer, but he or she just happens to have a somewhat different background. The fact that there are some projects which bundle Emacs preconfigured as a development environment for Common Lisp lowers the entry bar somewhat, but it is not always enough.

The second option is to use a freely available, proprietary, limited version of a commercial Common Lisp programming environment, be it Allegro or LispWorks. Despite the limitations, this option is viable for a beginner who wants to try some Common Lisp if he or she does not mind the limitations. There is always an option to move to another development environment after the initial learning phase.

From analysing the options above it becomes clear that there is a niche for an easy to install, open source Common Lisp implementation which provides a familiar environment for the most commonly used operating systems - Windows. The fact that Corman Lisp is one of the few implementations which have its own integrated development environment makes it a good candidate to fill the niche. Lowering the bar of entry is important, as it might help to attract more users to the Common Lisp. In my opinion, it is something which it desperately needs.

I should say that lowering the entry bar for a beginner is one of my main motivations behind contributing to the project.

It has a lot of educational value

One of the more interesting characteristics of the Corman Lisp is that it is incredibly compact, while it still provides all the major features of a Common Lisp. Its runtime, the Lisp kernel, is one of the most compact ones, yet it contains all the important parts of an efficient implementation of a dynamic programming language.

The fact that I was able to start making important changes to it in a relatively short time says a lot.

It has unique features

Although there are few very good Common Lisp implementations available, both open source and proprietary, Corman Lisp has some features which are not always available elsewhere:

It can be used to create dynamic link libraries (DLLs). It is possible to develop the plug-ins to the existing applications on par with C.
Corman Lisp kernel has the interfaces for integration into the other C++ applications as a scripting engine. It does not force you to write the whole application in Common Lisp.
A Corman Lisp's heap image can be dumped and restored without restarting the process. There are not many implementations which allow this.
It is possible to reuse C definitions directly from header files. It is very handy when building Lisp bindings to the C libraries.
Corman Lisp is one of the few implementations which feature its own integrated development environment.

The fact that Corman Lisp is 32-bit only limits its use cases. Nevertheless, it has a lot of interesting features which make it convenient in some cases where using other implementations is hard or even not possible.

It is a major achievement

Probably Common Lisp has the highest number of implementations to the number of users ratio. It contrasts to most of the other dynamic programming languages which typically have one reference implementation. I think that it is a result of being defined through a standard which thoroughly describes the programming language rather than a sign of the simplicity of the language itself.

Most of the implementations started as commercial products or their development was initially supported by a government because such a project typically requires a lot of effort and needs good financial support. This is a result of Lisp being a programming language which requires non-trivial programming techniques to be implemented efficiently. This fact alone makes any available Common Lisp implementation a valuable technical achievement.

Corman Lisp is not an exception here. It is a bit different though, as it was mostly developed by one person. Taking into consideration how much effort it takes to develop a usable Common Lisp implementation, it is a very valuable achievement as doing it this way surely needed the different approaches. A curious individual might learn a lot from studying it, doing so reveals a lot about how to individually develop a non-trivial project.

You might like a video in which Roger Corman reveals his motivation behind starting the project as well as provides some technical details about it.

Conclusion

After reading the information above you should understand why I think that maintaining Corman Lisp is important. I believe it is a good implementation which, as it was mentioned earlier, very compact and, thus, open for innovation and contributions. It surely has its downsides but it can be iteratively improved to a more usable state. It is also extremely easy to start working on.

While working on it I gained interesting experience as doing so required some knowledge which I had no chance to gain anywhere else. As it turns out, there is no need to be a programming rock star to do meaningful work on a non-trivial project. When I started working on the project I did not know Lisp very well and now I am only moderately better.

The fun continues and I am looking forward to what goes next!

A Handmade External Antenna For a Mobile Modem or Router

Wed, 03 Oct 2018 00:00:00 +0300

Recently I have completed my tiny DIY-project: an external antenna for a 3G modem/router so I decided to make a post to provide some additional details as well as to publish some photographs. You can find them at the end of the post.

According to my measurements, the antenna gives the gain around 8-9 Db. I believe it is a good result: maximal theoretical gain of such an antenna is 10.5 Db. I wanted to make a portable variant that I can take with me in a case of travelling.

The design of the antenna is well known here in Ukraine as well as in other ex-USSR countries. Its name is 'Kharchenko's antenna' ("антенна Харченко" in Russian, "антена Харченка" in Ukrainian), named after a Soviet engineer who invented it. The article with the original design was published in edition #11 of a magazine 'Radio' ("Радио" in Russian) in 1964. You can find the article on page 50. By design, it is an (omni)directional antenna. An almost the same design, with minor differences, was popularised in the English-speaking part of the Internet by Trevor Marshall, so it is often named after him there ('Trevor Marshall's antenna'). Another very common name for it is 'Bi-Quad antenna.'

Originally, it was designed for receiving TV-signal, but with the development of the technologies, the base design was adapted for the new standards, e.g. Wi-Fi, CDMA, 3G (WCDMA) and so on. Indeed, I can remember this antenna being used for receiving TV-signal when I was a child. To this very day, it is one of the most practical designs for many applications. Here, in ex-USSR, its variant is often used for boosting Wi-Fi signal (an example). I believe it is one of the best designs for a portable antenna, as it is possible to fit all the parts in a compact case.

One might wonder why I made an antenna for 3G in the times when the 4G is widely deployed? Well, here, in Ukraine, 4G has appeared only this year (yes, in 2018). 3G was introduced a couple of years earlier so it is more widely available: in the case, when you might need an external antenna, there are more chances to receive the 3G signal rather than 4G.

Here I shall quote William Gibson:

The future is already here — it's just not very evenly distributed.

I am not complaining though, as it is better to have something later than never.

If you are intrigued enough and want to make one for your own use, you have to do all the necessary calculations to make the antenna that is well tuned for a targeted radio frequency. Fortunately, there is an Android application, named 'Cantennator', which can do all the necessary calculations for you. By the way, the application does support other antenna designs too. A real gem for an antenna geek!

The author of the program has a site (in Russian) which is dedicated to different antenna designs for numerous applications. The information on this site was invaluable.

If you want to make an antenna for 4G, then you should know that 4G uses MIMO so you will need two similar antennas or one, which naturally supports MIMO. In this case, 'Patch-Yagi' antenna seems to be a better alternative, as you can connect two cables to them. Unfortunately, it is harder to make this one portable.

I did not plan to publish a detailed guide on making the antenna. I believe it is rather easy to understand by looking at the provided photographs. Moreover, I think this approach would stimulate your creativity. For your convenience, I provided the list of the main parts and materials, which I used, as well as some recommendations. Please keep in mind that I am not an expert on this topic though.

You will need:

Around 1 meter of thick copper wire (with the diameter of 2.5mm) for the vibrator. The aluminium one will do, but it is harder to solder.
A sheet of copper or aluminium for the reflector. I reused one from an empty aluminium can (from cider). In fact, any copper plated or aluminium plated material will do. If you have decided to follow my route, you might find this video very useful. I should note that reflector is not connected to the coaxial cable in any manner for this antenna. I have seen many similar antennas where it is, but, I believe, it is an error.
A set of connectors. I used the BNC-connectors, but I would advise you to get N-connectors instead because they are better suited for the task. Try to avoid cheap connectors. Good connectors are vital for an antenna.
Around one meter of 5Ohm coaxial cable. I used the thin RG-58 cable as I wanted the portable design, but ideally, you should use much thicker wire with the diameter at least 1cm, e.g. LMR-400 or RG-8. It especially makes sense if you want to make non-portable outdoors antenna. If you decided to use a thin cable for a portable variant, LMR-195 is better than RG-58. If you need a cable, which is much lengthier than one meter, use a thick cable due to huge signal loss.
A [[https://en.wikipedia.org/wiki/Patch_cable]pigtail]] to connect the cable to a receiving device, e.g., 3G/4G modem or router. For example, the one I got is BNC to TS-9, but it depends on the device and, obviously, your choice of connectors. You might end-up with making the pigtail too. Avoid long pigtails due to signal loss. Ideally, it should be around 15cm, no more.
A case, which suits your design. I adapted a thoroughly cleaned plastic can from the pickled herring.
Optionally, a stand. I adapted a small tripod for a camera or phone.

As about the cost, I am not sure that buying a manufactured one costs significantly more, especially if you are going to buy the high-quality components. I do not care, because I made one for fun and buying something like an antenna is, basically, a hit-and-miss process. Moreover, I would always prefer a handmade thing to the one manufactured in China. Even if the last one looks better, the handmade one is less deceitful.

(The images are clickable.)

Daemonize - a Tiny C Library for Programming the UNIX Daemons

Mon, 01 Oct 2018 00:00:00 +0300

Whatever they say, writing System-V style UNIX daemons is hard. One has to follow many rules to make a daemon process behave correctly on diverse UNIX flavours. Moreover, debugging such a code might be somewhat tricky. On the other hand, the process of daemon initialisation is rigid and well defined so the corresponding code has to be written and debugged once and later can be reused countless number of times.

Developers of BSD UNIX were very aware of this, as there a C library function daemon() was available starting from version 4.4. The function, although non-standard, is present on many UNIXes. Unfortunately, it does not follow all the required steps to reliably run a process in the background on systems which follow System-V semantics (e.g. Linux). The details are available at the corresponding Linux man page. The main problem here, as I understand it, is that daemon() does not use the double-forking technique to avoid the situation when zombie processes appear.

Whenever I encounter a problem like this one, I know it is time to write a tiny C library which solves it. This is exactly how 'daemonize' was born (GitHub mirror). The library consists of only two files which are meant to be integrated into the source tree of your project. Recently I have updated the library and realised that it would be good to describe how to use it on this site.

If for some reason you want to make a Windows service, I have a battle tested template code for you as well.

System-V Daemon Initialisation Procedure

To make discussion clear we shall quote the steps which have to be performed during a daemon initialisation (according to daemon(7) manual page on Linux). I do it to demonstrate that this task is more tricky than one might expect.

So, here we go:

Close all open file descriptors except standard input, output, and error (i.e. the first three file descriptors 0, 1, 2). This ensures that no accidentally passed file descriptor stays around in the daemon process. On Linux, this is best implemented by iterating through /proc/self/fd, with a fallback of iterating from file descriptor 3 to the value returned by getrlimit() for RLIMIT_NOFILE.
Reset all signal handlers to their default. This is best done by iterating through the available signals up to the limit of _NSIG and resetting them to SIG_DFL.
Reset the signal mask using sigprocmask().
Sanitize the environment block, removing or resetting environment variables that might negatively impact daemon runtime.
Call fork(), to create a background process.
In the child, call setsid() to detach from any terminal and create an independent session.
In the child, call fork() again, to ensure that the daemon can never re-acquire a terminal again.
Call exit() in the first child, so that only the second child (the actual daemon process) stays around. This ensures that the daemon process is re-parented to init/PID 1, as all daemons should be.
In the daemon process, connect /dev/null to standard input, output, and error.
In the daemon process, reset the umask to 0, so that the file modes passed to open(), mkdir() and suchlike directly control the access mode of the created files and directories.
In the daemon process, change the current directory to the root directory (/), in order to avoid that the daemon involuntarily blocks mount points from being unmounted.
In the daemon process, write the daemon PID (as returned by getpid()) to a PID file, for example /run/foobar.pid (for a hypothetical daemon "foobar") to ensure that the daemon cannot be started more than once. This must be implemented in race-free fashion so that the PID file is only updated when it is verified at the same time that the PID previously stored in the PID file no longer exists or belongs to a foreign process.
In the daemon process, drop privileges, if possible and applicable.
From the daemon process, notify the original process started that initialization is complete. This can be implemented via an unnamed pipe or similar communication channel that is created before the first fork() and hence available in both the original and the daemon process.
Call exit() in the original process. The process that invoked the daemon must be able to rely on that this exit() happens after initialization is complete and all external communication channels are established and accessible.

The discussed library does most of the above-mentioned initialisation steps as it becomes immediately evident that implementation details for some of them heavily dependent on the internal logic of an application itself, so it is not possible to implement them in a universal library. I believe it is not a flaw, though, as the missed parts are safe to implement in an application code.

The Library's Application Programming Interface

The generic programming interface was loosely modelled after above-mentioned BSD's daemon() function. The library provides two user available functions (one is, in fact, implemented on top of the other) as well as a set of flags to control a daemon creation behaviour.

The first one is:

pid_t daemonize(int flags);

The function might be considered the portable substitution to the BSD's daemon() function.

Its only argument is the bit-mask flags which control different aspects of the daemon creation.

DMN_DEFAULT - Create daemon with default daemonization parameters (close existing file descriptors, reset signal handlers, change the current directory to /, set umask to 0).
DMN_NO_CLOSE - Do not close existing file descriptors and do not redirect standard file descriptors to /dev/null.
DMN_KEEP_SIGNAL_HANDLERS - Do not reset signal handlers to their defaults.
DMN_NO_CHDIR - Do not change the current directory of the daemon to /.
DMN_NO_UMASK - Do not set umask to 0.

One can use bit-or operator | to pass multiple flags at once, as it is often the case in C programming language.

This function, generally, follows fork() semantics. By default, the function returns PID of the daemon process to the parent process (which starts a daemon) or 0 to the daemon process. On error it returns -1 - this value might be returned to both parent and daemon process. In this case, the errno value is set accordingly. In the case, if during the daemonization an error has occurred, the errno value is transparently transferred to the parent process.

The second function is:

pid_t rundaemon(int flags,
                int (*daemon_func)(void *udata),
                void *udata,
                int *exit_code,
                const char *pid_file_path);

This function is a mere wrapper on top of daemonize() which might check and create PID-file if desired. It is recommended to use this function instead of daemonize() whenever applicable.

As you can see, this function has more arguments than the previous one. These arguments are:

int flags - daemon creation flags, see above;
int (*daemon_func)(void *udata) - pointer to the function to be called after successful daemonization (the actual daemon body);
void *udata - pointer to be passed as the value in a call to daemon_func;
int exit_code - pointer to a variable which receives value returned by daemon_func;
const char *pid_file_path - full pathname to the PID-file. It is used for checking if the daemon is not already running (by checking if the file exists and locked) and is created if it does not exist. When daemon exits, this file is removed under normal conditions. This value might be NULL, in this case no checks performed and no file is created.

rundaemon() shares return values with daemonize() with one notable exception: it returns -2 if daemon is running.

A Complete Example

I believe that the discussion would be incomplete without a full example. It shows how to write a basic daemon using the above discussed library. The following example might be used as a template for your own daemon.

The daemon performs the following actions:

Runs in background and creates PID-file;
Sets up logging via syslog facility;
Handles SIGTERM and SIGHUP signals (using signalfd() and select());
Exits on SIGTERM signal.

It is implemented on top of the above-discussed rundaemon() in an event-driven manner and distributed alongside the library as example.c file.

#include <unistd.h>

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <errno.h>

#include <sys/types.h>
#include <sys/select.h>
#include <sys/time.h>
#include <sys/stat.h>
#include <sys/signalfd.h>
#include <fcntl.h>
#include <syslog.h>

#include "daemonize.h"

/* The daemon process body */
static int example_daemon(void *udata)
{
    int exit = 0;
    int exit_code = EXIT_SUCCESS;

    int sfd = -1;
    sigset_t mask;
    struct signalfd_siginfo si;

    /* open the system log */
    openlog("EXAMPLE", LOG_NDELAY, LOG_DAEMON);

    /* greeting */
    syslog(LOG_INFO, "EXAMPLE daemon started. PID: %ld", (long)getpid());

    /* create a file descriptor for signal handling */
    sigemptyset(&mask);
    /* handle the following signals */
    sigaddset(&mask, SIGTERM);
    sigaddset(&mask, SIGHUP);

    /* Block the signals so that they aren't handled
       according to their default dispositions */
    if (sigprocmask(SIG_BLOCK, &mask, NULL) == -1)
    {
        closelog();
        return EXIT_FAILURE;
    }

    sfd = signalfd(-1, &mask, 0);
    if (sfd == -1)
    {
        perror("signalfd failed");
        sigprocmask(SIG_UNBLOCK, &mask, NULL);
        closelog();
        return EXIT_FAILURE;
    }

    /* the daemon loop */
    while (!exit)
    {
        int result;
        fd_set readset;

        /* add the signal file descriptor to set */
        FD_ZERO(&readset);
        FD_SET(sfd, &readset);
        /* One could add more file descriptors here
           and handle them accordingly if one wants to build a server using
           event-driven approach. */

        /* wait for the data in the signal file descriptor */
        result = select(FD_SETSIZE, &readset, NULL, NULL, NULL);
        if (result == -1)
        {
            syslog(LOG_ERR, "Fatal error during select() call.");
            /* a low level error */
            exit_code = EXIT_FAILURE;
            break;
        }

        /* read the data from the signal handler file descriptor */
        if (FD_ISSET(sfd, &readset) && read(sfd, &si, sizeof(si)) > 0)
        {
            /* handle the signals */
            switch (si.ssi_signo)
            {
                case SIGTERM: /* stop the daemon */
                    syslog(LOG_INFO, "Got SIGTERM signal. Stopping daemon...");
                    exit = 1;
                    break;
                case SIGHUP: /* reload the configuration */
                    syslog(LOG_INFO, "Got SIGHUP signal.");
                    break;
                default:
                    syslog(LOG_WARNING, "Got unexpected signal (number: %d).", si.ssi_signo);
                    break;
            }
        }
    }

    /* close the signal file descriptor */
    close(sfd);
    /* remove the signal handlers */
    sigprocmask(SIG_UNBLOCK, &mask, NULL);
    /* write an exit code to the system log */
    syslog(LOG_INFO, "Daemon stopped with status code %d.", exit_code);
    /* close the system log */
    closelog();

    return exit_code;
}


int main(int argc, char **argv)
{
    int exit_code = 0;

    pid_t pid = rundaemon(0, /* Daemon creation flags. */
                          example_daemon, NULL, /* Daemon body function and its argument. */
                          &exit_code, /* Pointer to a variable to receive daemon exit code */
                          "/tmp/example.pid"); /* Full path to the PID-file (lock). */
    switch (pid)
    {
        case -1: /* Low level error. See errno for details. */
        {
            perror("Cannot start daemon.");
            return EXIT_FAILURE;
        }
        break;
        case -2: /* Daemon is already running */
        {
            fprintf(stderr,"Daemon already running.\n");
        }
        break;
        case 0: /* Daemon process. */
        {
            return exit_code; /* Return daemon exit code. */
        }
        default: /* Parent process */
        {
            printf("Parent: %ld, Daemon: %ld\n", (long)getpid(), (long)pid);
        }
        break;
    }

    return EXIT_SUCCESS;
}

Conclusion

The objective of the library is to hide all the trickery of programming a daemon so you could concentrate on the more creative parts of your application. I hope it does this well.

If you are not only interested in writing a daemon, but also want to make yourself familiar with the techniques which are used to accomplish that, the source code is available. Moreover, I would advise anyone, who starts developing for a UNIX environment to do that, as it shows many intricacies of programming for these platforms.

Fixing Emacs's FlySpell Mode for Texts with Multiple Alphabets

Thu, 09 Aug 2018 00:00:00 +0300

Recently I have encountered a problem in Emacs when using flyspell-buffer command to spell-check a text in Emacs. I tracked down the problem and decided to report the bug.

The problem was complicated and it was possible to reproduce it only for texts which use more than one writing system (e. g. Latin and Cyrillic) and only under some circumstances.

Emacs developers were very responsive and after few days Eli Zaretskii tracked down the root of the problem and provided a better fix which is going to be incorporated into Emacs 26.2.

If you happen to use an older version of Emacs and have to spell-check some multilingual texts often, you might want to consider redefining flyspell-external-point-words in your configuration file with the version, provided below.

(defun flyspell-external-point-words ()
  "Mark words from a buffer listing incorrect words in order of appearance.
The list of incorrect words should be in `flyspell-external-ispell-buffer'.
\(We finish by killing that buffer and setting the variable to nil.)
The buffer to mark them in is `flyspell-large-region-buffer'."
  (let* (words-not-found
         (flyspell-casechars (flyspell-get-casechars))
         (ispell-otherchars (ispell-get-otherchars))
         (ispell-many-otherchars-p (ispell-get-many-otherchars-p))
         (word-chars (concat flyspell-casechars
                             "+\\("
                             (if (not (string= "" ispell-otherchars))
                                 (concat ispell-otherchars "?"))
                             flyspell-casechars
                             "+\\)"
                             (if ispell-many-otherchars-p
                                 "*" "?")))
         (buffer-scan-pos flyspell-large-region-beg)
         case-fold-search)
    (with-current-buffer flyspell-external-ispell-buffer
      (goto-char (point-min))
      ;; Loop over incorrect words, in the order they were reported,
      ;; which is also the order they appear in the buffer being checked.
      (while (re-search-forward "\\([^\n]+\\)\n" nil t)
    ;; Bind WORD to the next one.
    (let ((word (match-string 1)) (wordpos (point)))
      ;; Here there used to be code to see if WORD is the same
      ;; as the previous iteration, and count the number of consecutive
      ;; identical words, and the loop below would search for that many.
      ;; That code seemed to be incorrect, and on principle, should
      ;; be unnecessary too. -- rms.
      (if flyspell-issue-message-flag
          (message "Spell Checking...%d%% [%s]"
               (floor (* 100.0 (point)) (point-max))
               word))
      (with-current-buffer flyspell-large-region-buffer
        (goto-char buffer-scan-pos)
        (let ((keep t))
          ;; Iterate on string search until string is found as word,
          ;; not as substring.
          (while keep
        (if (search-forward word
                    flyspell-large-region-end t)
            (let* ((found-list
                (save-excursion
                  ;; Move back into the match
                  ;; so flyspell-get-word will find it.
                  (forward-char -1)
                              ;; Is this a word that matches the
                              ;; current dictionary?
                              (if (looking-at word-chars)
                      (flyspell-get-word))))
               (found (car found-list))
               (found-length (length found))
               (misspell-length (length word)))
              (when (or
                             ;; Misspelled word is not from the
                             ;; language supported by the current
                             ;; dictionary.
                             (null found)
                 ;; Size matches, we really found it.
                 (= found-length misspell-length)
                 ;; Matches as part of a boundary-char separated
                 ;; word.
                 (member word
                     (split-string found ispell-otherchars))
                 ;; Misspelling has higher length than
                 ;; what flyspell considers the word.
                             ;; Caused by boundary-chars mismatch.
                             ;; Validating seems safe.
                 (< found-length misspell-length)
                 ;; ispell treats beginning of some TeX
                 ;; commands as nroff control sequences
                 ;; and strips them in the list of
                 ;; misspelled words thus giving a
                 ;; non-existent word.  Skip if ispell
                 ;; is used, string is a TeX command
                 ;; (char before beginning of word is
                 ;; backslash) and none of the previous
                 ;; conditions match.
                 (and (not ispell-really-aspell)
                                  (not ispell-really-hunspell)
                                  (not ispell-really-enchant)
                  (save-excursion
                    (goto-char (- (nth 1 found-list) 1))
                    (if (looking-at "[\\]" )
                    t
                      nil))))
            (setq keep nil)
                        ;; Don't try spell-checking words whose
                        ;; characters don't match CASECHARS, because
                        ;; flyspell-word will then consider as
                        ;; misspelling the preceding word that matches
                        ;; CASECHARS.
                        (or (null found)
                (flyspell-word nil t))
            ;; Search for next misspelled word will begin from
            ;; end of last validated match.
            (setq buffer-scan-pos (point))))
          ;; Record if misspelling is not found and try new one
          (cl-pushnew (concat " -> " word " - "
                       (int-to-string wordpos))
                              words-not-found :test #'equal)
          (setq keep nil)))))))
      ;; we are done
      (if flyspell-issue-message-flag (message "Spell Checking completed.")))
    ;; Warn about not found misspellings
    (dolist (word words-not-found)
      (message "%s: word not found" word))
    ;; Kill and forget the buffer with the list of incorrect words.
    (kill-buffer flyspell-external-ispell-buffer)
    (setq flyspell-external-ispell-buffer nil)))

Alternatively, you can put the function definition into separate file (flyspell-fix.el) inside your Emacs configuration directory (~/.emacs.d) and load it only for Emacs versions which are older than 26.2.

After creating flyspell-fix.el file, as mentioned above, add the following lines into your Emacs's configuration file:

;; fix flyspell for older versions of Emacs.
(when (or (< emacs-major-version 26)
          (and (= emacs-major-version 26)
               (< emacs-minor-version 2)))
  (load (concat user-emacs-directory "flyspell-fix.el")))

dumpmem() - a Function to Dump an Area in RAM in Hexadecimal Format

Wed, 08 Aug 2018 00:00:00 +0300

Today I want to share with you some code which helped me to track down a problem in a rather complex networking VoIP application. I believe it might be useful in other scenarios as well.

Some time ago I debugged a networked application. The problem seemed to be in the handling of one type of network packets and two versions of the application behaved differently under similar circumstances. There is one known issue with debugging networked applications - timing might be very important and, in some cases, there is no much time to diagnose raw data which was received via networking socket in a debugger. More than that, the format of the data was binary and, thus, not human readable. Using network sniffing software (like Wireshark) was not an option, as parts of the message contained encrypted information.

To verify my hypothesis without interfering much with the application during runtime, I decided to dump contents of the parts of the message to a log file in hexadecimal format and analyse the data later. The format closely resembles one, used by HEX-editors, as you will see. In fact, I adopted most of the source code from a HEX-viewing utility which I wrote as a university assignment a long time ago.

Here is the source code of a function, which dumps contents of a part of random access memory in hexadecimal format:

void dumpmem(FILE *out, const void *ptr, const size_t size)
{
    const size_t BYTES_PER_LINE = 16;
    size_t offset, read;

    uint8_t *p = (uint8_t *)ptr;
    const uint8_t *maxp = (p + size);

    if (out == NULL || ptr == NULL || size == 0)
    {
        return;
    }

    for (offset = read = 0; offset != size; offset += read)
    {
        uint8_t buf[BYTES_PER_LINE];

        for (read = 0; read != BYTES_PER_LINE && (&p[offset + read]) < maxp; read++)
        {
            buf[read] = p[offset + read];
        }

        if (read == 0)
            return;

        fprintf(out, "%.8x: ", (unsigned int)offset);

        /* raw data */
        for (size_t i = 0; i < read; i++)
        {
            fprintf(out, " %.2x", buf[i]);
            if (BYTES_PER_LINE > 8 && BYTES_PER_LINE % 2 == 0 && i == (BYTES_PER_LINE / 2 - 1))
                fprintf(out, " ");
        }

        /* ASCII */
        if (read < BYTES_PER_LINE)
        {
            for (size_t i = read; i < BYTES_PER_LINE; i++)
            {
                fprintf(out, "  ");
                fprintf(out, " ");
                if (BYTES_PER_LINE > 8 && BYTES_PER_LINE % 2 == 0 && i == (BYTES_PER_LINE / 2 - 1))
                    fprintf(out, " ");
            }
        }
        fprintf(out, " ");
        for (size_t i = 0; i < read; i++)
        {
            if (buf[i] <= 31 || buf[i] >= 127) /* ignore control and non-ASCII characters */
                fprintf(out, ".");
            else
                fprintf(out, "%c", buf[i]);
        }

        fprintf(out, "\n");
    }
}

As one could see, it accepts three parameters:

out - output file stream;
ptr - pointer to data;
size - data size.

There is some room for optimisation, but I think that it is not worth it.

If you wonder how the output of this function looks like, here is a short example of an H.225 RAS message:

00000000:  03 00 00 ab 08 02 18 af  05 04 04 88 18 a0 a5 28 ...............(
00000010:  06 76 63 34 30 30 00 6c  04 81 36 30 31 7e 00 8b .vc400.l..601~..
00000020:  05 20 b8 06 00 08 91 4a  00 06 02 01 00 93 44 04 . .....J......D.
00000030:  00 76 00 63 00 34 00 30  00 30 22 c0 26 00 00 02 .v.c.4.0.0".&...
00000040:  0e 59 65 61 6c 69 6e 6b  20 56 43 34 30 30 00 00 .Yealink VC400..
00000050:  0b 33 30 2e 31 30 2e 30  2e 32 37 00 00 00 01 40 .30.10.0.27....@
00000060:  03 00 61 00 73 00 72 00  76 00 c0 a8 29 9c 06 b8 ..a.s.r.v...)...
00000070:  00 dc 27 ec f9 6b ef e5  11 94 0e 00 15 65 86 fc ..'..k.......e..
00000080:  9f 00 5d 0d 80 07 00 c0  a8 5a 0e 06 b8 11 00 82 ..]......Z......
00000090:  27 ec f9 6b ef e5 11 94  0e 00 15 65 86 fc 9f 01 '..k.......e....
000000a0:  00 01 00 01 00 01 00 02  80 01 00                ...........

Off course, the discussed debugging technique is not the panacea and should be used only in very specific cases, as, I believe, one could obtain similar results using modern debugging tools. But anyway it helped me to solve a problem in short time, so I decided to share this information here. It is not a tool for everyday use, but still, I think, it is a trick worth to know about.

Persistent Remote Terminal Session With TMUX

Thu, 05 Apr 2018 00:00:00 +0300

The first thing I install on every UNIX machine on which I have enough administrative rights is tmux (terminal multiplexor) - I feel uncomfortable to connect to it via SSH otherwise. For example, you could encounter some problems with your Internet connection - any sudden disconnect might disrupt a lengthy operation like backup which might have been initiated from within the remote terminal session. I want my remote session to be persistent.

I used to have a habit to start a tmux session right after I have connected to a remote server via SSH. Recently I realised that this action can be automatised. I ended up with the following code-snippet which might be placed into your shell initialisation file (e.g. .bashrc for bash):

# Attach to the tmux session on login via SSH.
if [ ! -z "$SSH_CLIENT" ] && [ -z "$DESKTOP_SESSION" -a -z "$TMUX" ] ; then
    tmux attach -t remote || tmux new -s remote
fi

The above-mentioned snippet automatically tries to connect to an existing tmux session named remote or start a new one when logged-in via SSH. It contains some additional checks, which should be discussed:

There is a check for empty DESKTOP_SESSION environmental variable to avoid connecting to the tmux session from within remote X Session initiated over SSH;
There is a check for empty TMUX environmental variable to avoid connecting to the tmux session recursively.

I believe that being able to automatically reconnect to an existing terminal session is very convenient. I hope you will find it useful too.

Launch!

Fri, 30 Mar 2018 00:00:00 +0300

So, here goes yet another technical blog. The need to make one emerged after the realisation that I miss technical writing a lot since I left a university. Moreover, I hope that it will help me to improve my written English.

Although I have no particular agenda, you might expect from me writing about computer science, technology, and programming. I am not limiting myself to these topics though. In general, I am going to keep writing about whatever seems interesting to me at the moment or revisit some of my older stuff if there is nothing more entertaining that catches my attention.

Hope to see you soon!