Category Archives: Software Engineering

How many values are errors in Go?

This post is written as a follow-up on Go’s Error Handling Sucks – A quantitative analysis upon the request of Herb Sutter who asked for a rough percentage of function calls that return error information to all function calls.

For this purpose, I have used the following regex for capturing function calls that return an error:

[a-zA-Z0-9_]*[eE]rr(or)? *:?= *([a-zA-Z0-9_]*\.)*[a-z-A-Z0-9_]+\(

and the following regex for capturing any function call:

([a-zA-Z0-9_]*\.)*[a-z-A-Z0-9_]+\(

The first regex relies on the convention that error is the last (or the only) value of any function that returns error information.

The second regex has the following shortcomings which should be understood:

  • Casts are mistaken as function calls.
    E.g. ENFILE = syscall.Errno(0x17)
  • Counting built-in “functions” such as len, append etc might be undesirable.
  • Function definitions, imports, and consts are confused as function calls, but I filtered them in my calculations.

Lastly, I have used the following commands to count:

ag --go --stats-only "[a-zA-Z0-9_]*[eE]rr(or)? *:?= *([a-zA-Z0-9_]*\\.)*[a-z-A-Z0-9_]+\\("

ag --go -v "func|const \\(|import \\(" | ag --stats-only "([a-zA-Z0-9_]*\\.)*[a-z-A-Z0-9_]+\\("

Here are the results:

# of Calls Returning error# of Function CallsRatio
Kubernetes83,442740,38511.27%
Go27,741482,5465.75%
Moby (Docker)37,044310,73611.92%
Cockroach DB24,498254,9269.61%
Mattermost (server)22,737214,75610.59%
etcd13,761135,95710.12%
Syncthing2,43625,4729.56%
Hugo1,87623,9837.82%
Helm1,92711,44316.84%
Gin2425,3074.56%
TOTAL215,7042,205,5119.78%

For the record, here are the repositories used and their respective commits:

Go’s Error Handling Sucks – A quantitative analysis

Go’s error handling sucks because it forces its developers to propagate errors up the callstack manually, whereas you could CATCH and THROW errors since 1972 using MacLisp according to Wikipedia.

Some claim that Go’s way of handling errors encourages developers to actually handle the damn errors instead of letting exceptions float freely in the hope of someone else handling it. I believe, on the contrary, the way we currently handle errors leads to visual satiation where your screen is filled with if err != nil { return ... } blocks as you manually propagate errors. It has become a masochistic cult where veterans are trying to convince newcomers that this is actually better.

Another shortcoming of Go’s errors is that they are extremely primitive. error is an interface with a single method Error() that returns a string intended for humans. That’s it. Callstack or any other debugging information is missing by default, unless you choose a third-party package such as pkg/errors and Wrap() your errors as you propagate them. The standard library doesn’t encourage sane behaviour either, forcing developers to either string-search or cast errors in undocumented ways.

To prove how inefficient this is, I’ll grep the source code of Go’s flagship projects for the following snippet:

if [...; ] err != nil {
    return ...
}

which is nothing but simply propagating the error up the callstck (i.e. return the error to the caller as soon as you encounter an error); something a compiler or a runtime would do if we were using a more modern language using the exceptions mechanism. For this purpose, I’ll use the following regex:

err != nil {\n\t*return

and multiply the number of times this occurs with 3, since it takes three lines of code.

# Lines of Go Code# Lines of Code for Manual PropagationRatio
Kubernetes2,912,66343,5874.49%
Go1,606,8297,7151.44%
Moby (Docker)1,013,08718,7385.55%
Cockroach DB776,34012,2384.73%
etcd444,2914,7263.19%
helm40,0567585.68%

In conclusion, 3.88% of the Go code written (excluding comments and blanks) consist of manual error propagation, which is problematic.

Even Russ Cox admits it.

pydis – A redis clone in Python 3 to disprove some falsehoods about performance

I developed pydis in two days to disprove some of the falsehoods about performance and optimisation regarding software and interpreted languages in particular.

Unfortunately many programmers, due to their lack of experience, of some knowledge of computer architecture(s), or of an in-depth understanding of the task they are given, spend countless hours by making life harder for themselves in the name of marginal performance gains, often trading many other conveniences (such as type safety, garbage collection, etc) too.

pydis is < 250 lines of idiomatic Python code, providing a subset of redis’ functionality for which there are official benchmarks, and yet it’s ~60% as fast as Redis measured in number operations per second.

Remember, man-hours are more expensive that cpu-time.

https://github.com/boramalper/pydis

Continue reading

Dropping root Privileges Permanently on Linux in C

If you would like to drop root privileges permanently on Linux using C:

#define _GNU_SOURCE  // for secure_getenv()


int drop_root_privileges(void) {  // returns 0 on success and -1 on failure
	gid_t gid;
	uid_t uid;

	// no need to "drop" the privileges that you don't have in the first place!
	if (getuid() != 0) {
		return 0;
	}

	// when your program is invoked with sudo, getuid() will return 0 and you
	// won't be able to drop your privileges
	if ((uid = getuid()) == 0) {
		const char *sudo_uid = secure_getenv("SUDO_UID");
		if (sudo_uid == NULL) {
			printf("environment variable `SUDO_UID` not found\n");
			return -1;
		}
		errno = 0;
		uid = (uid_t) strtoll(sudo_uid, NULL, 10);
		if (errno != 0) {
			perror("under-/over-flow in converting `SUDO_UID` to integer");
			return -1;
		}
	}

	// again, in case your program is invoked using sudo
	if ((gid = getgid()) == 0) {
		const char *sudo_gid = secure_getenv("SUDO_GID");
		if (sudo_gid == NULL) {
			printf("environment variable `SUDO_GID` not found\n");
			return -1;
		}
		errno = 0;
		gid = (gid_t) strtoll(sudo_gid, NULL, 10);
		if (errno != 0) {
			perror("under-/over-flow in converting `SUDO_GID` to integer");
			return -1;
		}
	}
	
	if (setgid(gid) != 0) {
		perror("setgid");
		return -1;
	}
	if (setuid(uid) != 0) {
		perror("setgid");
		return -1;	
	}

	// change your directory to somewhere else, just in case if you are in a
	// root-owned one (e.g. /root)
	if (chdir("/") != 0) {
		perror("chdir");
		return -1;
	}

	// check if we successfully dropped the root privileges
	if (setuid(0) == 0 || seteuid(0) == 0) {
		printf("could not drop root privileges!\n");
		return -1;
	}

	return 0;
}

I hope this would save you some time so that you won’t spend an hour like me, trying to find a proper solution!

The code is provided “as is”, without warranty of any kind, express or implied.

Sources

c – Dropping root privileges – Stack Overflow

man sudo (Sudo Manual)

Using LLVM’s IR for semi-compiled packages

As many other things that comes into my mind between 1-3 AM, this is also a bit blurry.

I was thinking about the possibility of utilizing LLVM’s Intermediate Representation to pre-compile packages into an intermediate language which is high level enough, and yet with its dense bitcode representation, can easily shorten compile times by reducing the time that is spent on parsing, analyzing and intermediate code generation.

Distros such as Gentoo can benefit from to a great extent I believe, but as I don’t know how their system works, I can not elaborate about the details. Also any open source project, that doesn’t provide pre-built packages, can at least use this method to shorten build times of their users.

I’ll think about it, to come up with an idea.