Recently I deactivated my (consulting) Twitter account.
For posterity, I made an archive of its contents available.
I just came across this article via the 2020-02-08 O’Reilly Programming Newsletter:
While I’ve greatly enjoyed — and become increasingly productive in — Go, using it mainly to enhance Joker to automatically include Go’s standard library and (more recently) start up more quickly (about which I plan to write another blog post soon) — I’ve been planning to look into Rust as a possible new low-level systems-programming language to replace C in my arsenal.
Would Joker (or a similar fast-startup, low-overhead, Clojure interpreter) be worth considering re-implementing in Rust?
I plan to look into startup-time performance for Rust programs soon!
I did my own research on this a couple of years ago, but just now discovered this helpful GitHub repo that has some interesting data:
The repo goes back as far as 9 years, and it isn’t obvious how recent the data is (though the version info provided with each language implementation is helpful).
But since the repo provides the ability to reproduce results on one’s own, it seems like a valuable resource.
In last November’s post, I discussed what was then called
gostd2joker, and have since renamed
gostd, which is a fork of Joker. Joker itself is a small, single-threaded linter and interpreter of a subset of Clojure. My fork seeks to automate creation of wrappers around much (if not all) of the Go standard library, and perhaps other arbitrary packages in the future, so they are accessible (even if via fairly primitive, low-level mechanisms that aren’t idiomatic Clojure) via running Joker code.
Though I put that project aside for several months earlier this year, while pursuing other fields of endeavor, I returned to it a few months ago and have made substantial progress. Compare the namespaces provided by canonical Joker to those provided (though incompletely, in most cases) by my fork.
Besides that work, the canonical version of Joker continues to improve at a reasonable pace, and is increasingly useful as a scripting language.
In a recent podcast, Joker’s author, Roman Bataev, discusses Joker’s history, capabilities, and potential futures, including a brief shout-out to my fork at around the 15:00 mark:
I continue to enjoy working on this (currently unfunded) project, due to its heady mix of Clojure (a well-designed Lisp variant), Go (a well-designed imperative language), and automated code generation.
As much fun as coding can be, I prefer to write code that generates code.
My most recent effort is gostd2joker, which collects information on Go’s standard library (the packages provided in its source tree) and generates the Go and Clojure code to provide access to some of those APIs within Joker, a Clojure interpreter that is written in Go. While there remain substantial limitations in terms of which APIs are eligible for transformation, the resulting list of converted packages is noticeably larger than that for “vanilla” Joker.
The result should be quite useful, not only in my own research and development, but potentially to others who might find a quick-starting, low-overhead scripting language, with full access to the Go runtime, to be a formidable tool in their scripting arsenal.
Can you envision using such an enhanced version of Joker? If so, how?
I’ve started deleting users who haven’t commented on, or posted, anything, to get rid of the many likely-spam/bot users.
The bulk-delete tool I’m using doesn’t let me filter on whether a user has a profile image, and it appears to have already deleted at least two legit-looking subscribers. So I’m holding off deleting more for now.
Please login and comment/post on occasion, at least for awhile, as I clear out inactive users. (I’ve added a plugin to reduce new-user registrations from known spammers as well.)
Thanks for your cooperation!
How can organizations that depend on free-software components ensure that they are maintained and available, which requires (first and foremost) funding component developers? This article explains some different approaches to that problem.
Over the past several weeks, I turned my attention to assessing the performance of various Lisp implementations, focusing on suitability for short scripts, launched as processes, very frequently as is consistent with the traditional Unix model and exploited by messaging systems such as qmail.
Here are some interesting results on the use of Go, versus C, as an implementation language for a Lisp interpreter (Clojure in particular, though that’s not involved in these results):
tl;dr: Go is. Really, really fast, at least on this simple code and its few tests. It’s certainly not enough slower than C to justify implementing a Lisp interpreter in C, especially when there’s already a decent, albeit young, Clojure interpreter written in Go.
There’s plenty to be said (and has already been said) about Go versus C. I found debugging Go code to be challenging, as the use of GDB was discouraged, and Delve does not meet my requirements for easily debugging such low-level code. But the packaging system (the go tool) is fairly well-thought-out, easy to come up to speed on, and surprisingly free of warts.
Great job, Go people!
Interesting video on software-development history:
Of the four major performance characteristics, or dimensions — processor (CPU) speed, memory (storage), bandwidth, and latency — latency has seen only modest improvements over the past several decades of modern computing, while the other three have experienced growth substantially described by Moore’s Law.
That is, the speed, storage, and bandwidth offered by computing (information) systems have improved substantially year over year for many years. Meanwhile, the limiting factor for latency — the speed of light — has experienced no improvement whatsoever, nor is there any such improvement (increase in the speed of light or an alternative means of communication not limited by it) anticipated in the near future, short of a stunning breakthrough in physics. Most improvements in latency have been in the form of modest percentages “on the edges” (of electronic or light-based transmission) as pertinent circuitry has leveraged the benefits in those other three dimensions of performance.
As a “green-field” starting point for information-systems architecture, today’s state of the art would suggest that systems be optimized to minimize latency, first and foremost, to the extent those systems encompass large-scale networks, even if at the expense of increased demands for processing speed, storage, and bandwidth.
For example, Verizon Enterprise’s IP Latency Statistics suggest, as of January 2018, latencies ranging from 8ms to nearly 300ms, depending on the endpoints.
A system that requires N round trips to perform an action can therefore expect performance no better than N*300ms in the worst case documented on that page, N*8ms in the best case.
Such “round trips” include not only overt actions performed by the application, but actions performed at lower levels in the stack in order to carry out those high-level actions. E.g. setting up an outbound TCP connection takes several round trips in and of itself.
Optimizing various endpoints, or across individual stack levels, cannot improve beyond this minimum threshold, unless the system itself is optimized (essentially, re-architected) in some fashion.
While keeping latencies low is well-understood at various levels, or slices, of the computing stack (see Understanding Bandwidth and Latency for just one example), these efforts can fall well short of the full potential of optimizing for low latency as it pertains throughout the stack, including when done via Architecture.
Here, viewing Architecture‘s goal as “making fundamental structural choices which are costly to change once implemented”, the protocols used to implement the communications must necessarily be optimized, via their selection and/or design, for latency — as must any other aspects of the system where performance is a consideration. (Note that optimizing the design of a protocol is distinct from optimizing its implementation, similarly to how choosing an algorithm, based on its performance characteristics, is distinct from optimizing the implementation of that algorithm.)
Thinking back in time prior to today’s theoretical “green-field” starting point, it becomes easier to understand why and how network protocols (including TCP, SMTP, HTTP, and so on) were designed to meet performance requirements that did not prioritize latency above all. In particular, architects and designers had to cope with systems and networks of comparatively limited processing power, storage, and bandwidth — but seemingly “instantaneous” communication (at near the speed of light). Naturally, they chose to ensure their protocols would be realizable by computing machinery with power that was severely limited compared to today’s.
Scaling up systems built on top of those legacy protocols, to support an increasingly widespread online population — of both man and machine — is proving to be increasingly challenging, for reasons of latency at least, nevermind the ongoing challenges presented by sheer complexity, security issues, and so on.
That’s why it’s important for new designs to start out architected to avoid high-latency pitfalls, perhaps even to the extent of avoiding those legacy protocols in favor of lower-latency alternatives (such as UDP and datagrams rather than TCP).
It’s also worth considering whether a “21st-Century” Internet, architected for low latency, is worth pursuing as a “Beyond IPv6” offering.
What might a protocol that optimizes for latency look like? I’m working on a Proof of Concept, and hope to have something to show for it in the near future.
Please let me know what experiences and thoughts you have on optimizing for latency!