I started learning a third language in June. There were quite a few options, but I insisted on the principle of project-based learning, and that led to the decision: learn Tcl/Tk and write a GUI app for yt-dlp, a web video downloader.
In fact, I decided it was time to go one step further in the spirit of letting a real project guide the journey of learning. With Elm and Rust, I actually started in the conventional way, namely with books. Luckily, both official books (the Elm Guide, and TRPL) were very good, so I didn't suffer to say the least; but that meant it was still quite like school: you are provided with a systematic knowledge base, so that things are easy to follow. But one of my main realizations from school is that true internalization doesn't come from just being taught well. And after doing 5 projects on my own, I think I've found the missing ingredients: real questions, and real practices, and the keyword is of course real. And if you have a real project, then you've got both for free.
But what makes a project real? I think the criterion is mostly psychological. It's not about whether your project turns out to be "production-grade" or "industrial-strength", the essential factor is whether you think it serves as a satisfying solution to a real problem you're motivated to tackle.
And yes, "motivation is everything", as Feynman said in his autobiography, paraphrasing. One has to feel truly connected to a problem in order to gather the necessary "activation energy" to start the creative mode, and one has to be creative in order to solve real problems (otherwise there would be existing solutions already), and I think staying in the creative mode long enough is the key to true internalization.
So this time, I got into the project before knowing much about Tcl at all. Granted, this would probably not work if the language is modeled after some deep theory. But Tcl is a quintessential GTD language, "command" is how you do things. I heard the syntax is extremely minimal, so any "complexity" would be compartmentalized within a particular command.
My overall experience report: I did not expect I could get the GUI to a satisfying state of functionality and visual design without using any third-party libraries, that is, the features contained in the core are so well selected that they cover most use cases out of the box, this is very empowering.
That led to a probably bad hypothesis: could this well-designed nature of a language actually make it less popular, or rather, the community less active? Because people can just use it as is and get productive, without having to struggle and build "frameworks" to overcome the issues of the language, and therefore build communities, often competing ones, around some design patterns external to the core?
For the record, I still read John Ousterhout's book (extremely well-written), just not sequentially, since the point of this experiment is to guide the learning by actual problems to solve in a real project. And I think it works. By far, about 20% of total project time is spent on learning, and about 60% on implementing functionality (plus 10% on improving existing functionality). This 20% is expectedly much higher than the fraction in my Elm and Rust projects (2-10%).
With this approach, I'm also unburdened by the compulsion to learn things prematurely, by that I mean learning without a true motivation (a true connection to a real problem). I first heard of Tcl when looking at Redis's language composition on GitHub. Then I went read Sanfilippo's Tcl the Misunderstood. Knowing that the metaprogramming power of Tcl lies in things like upvar and uplevel, I would have focused on them during the initial learning phase. But this time, I wanted to wait and see how long I could get away with not knowing the advanced stuff at all. Turns out I could go quite far, given that this project is about making a GUI app - there's no need to design a DSL (yet). The point is, this new pedagogy keeps me laser-focused on the project, instead of trying to be knowledgeable. There is no "A" to get, but a product to deliver.
Tcl
The objective was very straightforward: make a GUI wrapper for yt-dlp to facilitate running and monitoring multiple download tasks in parallel. Previously the workflow had been just opening a dozen of console windows and entering commands into each, effectively keeping a "pool" of prompts. Monitoring all the tasks and retrying failed ones were quite awkward. A GUI driver seemed to be a natural fit for the scenario.
So I knew the core of the GUI app was the redirection of input and output to and from the CLI. I needed to make sure Tcl/Tk was suitable for such tasks as "streaming" the standard output/error of a process to some GUI element in realtime, non-blocking. Therefore, I didn't bother digging into the Tk manual first to make a mock GUI before this essential functionality could be demonstrated.
It turned out that Tcl was fully capable of the task, without involving any third-party libraries. You can choose between blocking (default) and non-blocking as the mode of reading or writing for a channel (which could be either a process or a file), and even switch the mode on the fly anytime before you close the channel (e.g. I read that in order to get the process's exit code, you need to set the channel back to blocking mode before you close it).
Some subtle complexity seemed unavoidable. Tcl by default does automatic translation of line-break characters according to the platform, but in my use case (I picked Windows 7 for this project), that mechanism "scrambles" the signal (the leading carriage return character of a line) I rely upon to differentiate a regular line of output from a "transient" progress line, which reports the completion percentage at a given instant, along with downloaded bytes, ETA, and so on. Therefore, I must disable such mechanism of conversion to preserve the original line-breaks. But I understand Tcl's perspective, as it very much strives for cross-platform compatibility that works out of the box. On the other hand, nowadays, as programmers on different OS platforms collaborate regularly, people seem to converge to some universal convention instead of sticking to each platform's own. This does help reduce unnecessary code complexity and thus bugs, e.g. yt-dlp uses the same line-breaks regardless of which platform it runs on.
String and list APIs are well designed: they are small and "orthogonal", and that's key! A large, high-redundancy API paralyzes people, esp. newbies. Every time I look at Rust's Result and Option docs, I have an entirely different feeling from reading Tcl or Elm. In a way, Rust's design choice does serve savvier people, because with this much coverage they no longer have to write their own set of helpers for common patterns (i.e. the "lodash" kind of package). And indeed, that's why it's a good thing that we have both "big" languages like Rust (Rust is very USA), and small ones like Elm and Tcl.
Substitution is what takes the most getting used to. As in most newer languages, variable substitution is completely transparent/automatic, we don't even think about it (the occasional shell scripting is the only place where I have to deal with it). But for Tcl, I think it's well justified, for it's what it takes to be a truly "string-oriented" language - everything by default is a string, unless you specify other meanings using special symbols. A simpler syntax that has less structure inevitably has to expose some extra concern to the users, a form of trading verbosity for uniformity. And uniformity is a wonderful thing for expressing ideas in a simple and robust way, while verbosity is something we can get used to and then it cognitively disappears. But note that by verbosity I do not mean it takes more lines of code in Tcl to get something done: on the contrary, Tcl is well-known for its expressive power (the entire GUI program is 400 LOC with comments), here I'm just referring to the extra symbols to indicate substitution that can be made invisible in other languages with a more complex syntax rule-set.
Tk
Tk isn't "reactive", and it's said to be very "object-oriented" (widgets, widget classes, widget commands), and my overall impression is that Tk is very "imperative", as opposed to "declarative" (I suppose reactive implies declarative). For example, if you create the parent widget after its children, then the children will be obscured, unless you "raise" their stacking order next. It's not surprising really, since a Tcl program is just a sequence of commands, but still that is a very different experience compared to working in the Elm Architecture. It's very interactive though - in a console, you can enter a command to destroy any existing widget in the running GUI, and then enter another to recreate one.
One might imagine that this can easily lead to spaghetti code, but given the scope of this project, so far I find it quite manageable, and the widget naming rule helps the organization quite a bit - a widget must be named/identified by its full path in the entire widget tree. For instance, if you are to create two buttons within some container, then they must be named .container.button1 and .container.button2, where the leading dot kind of signifies the root widget. This one, whenever you refer to a widget, you automatically know where it is in the hierarchy. But I guess then the drawback is the verbosity when working with a very deep widget tree. I think this can often be mitigated though - if you're manipulating a subtree of widgets deep down, then simply extract the long common prefix.
Layout
Geometry managers - grid and pack, are very well designed, versatile and intuitive. They are what people nowadays refer to as "layout", and the most popular model or concept is rows and columns, and indeed that's what the "grid" geometry manager is all about. However, after learning about the "pack" command, I found that it often produced significantly simpler code compared to using grid, esp. when trying to configure the "auto-fill" property. For instance, with "pack", often a combination of -fill and -expand in a one-liner will suffice, but with "grid", you typically need not only -sticky for the initial children arrangement, but also a separate columnconfigure -weight command subsequently to enable widget auto-resizing. For completeness, here's some sample code:
pack .g.c -expand 1 -fill both
grid .g.c -sticky nsew
grid columnconfigure .g .g.c -weight 1
grid rowconfigure .g .g.c -weight 1
For this reason, I find it suboptimal that in Ousterhout's book, "grid" is presented before "pack", and I was left with the impression that "grid" should generally be the default, go-to option.
The packer is easy to use in simple cases where a single row or column is needed, but it is also capable of complex layouts. Although you can achieve any arrangement of widgets with either the packer or the gridder, complex layouts are usually easier to create with the gridder.
The above statement is in theory correct, that grid is more generally powerful and expressive, but in practice, I find things are rather different:
Most GUI layouts today can be decomposed into single rows and columns, recursively. By that I mean, if you start from the top level window, it is typically a single column, with say 2 sibling elements along a single axis (i.e. the title bar, and the main container). And as you keep doing the decomposition analysis down the widget tree, you will find that at any given level, it is almost always possible to define a list of siblings that are arranged along a single axis. And this recursive compositions of one-dimensional widget arrangement can produce arbitrarily complex layout overall. Which is why the aforementioned rule-of-thumb, "The packer is easy to use in simple cases where a single row or column is needed", can be misleading. It didn't mean to apply only to cases of a single row or column of leaf nodes!
I said "almost always", didn't I? So when do we have to use grid instead of pack? When there is no way to correctly define a one-dimensional arrangement of siblings at a given level. The canonical example is a true grid, namely a table with at least two rows and two columns. Note that such a grid cannot be correctly decomposed into two levels in the hierarchy, for instance, a 2x2 table, which contains 4 siblings, is not the same as 2 siblings (each a 1x2 row or column) stacked next to each other, because in the later case, the two children of one sibling are independent from those of the other, while in the former, all the four children are geometrically coupled!
To be more specific, with a real table, the following property holds: all the cells of the same row have the same height, and all the cells of the same column have the same width.
Now if you redefine the layout by breaking a true two-dimensional grid into two hierarchical levels, as either a row of columns or a column of rows, then you will lose the above property of geometric coupling. Indeed, one of the few frustrations I have with Elm-UI was, because it discards the native HTML implementation of a table and essentially emulates one as a row of columns, if one column containers larger cells than those of another (e.g. larger font or padding), then the row borders of these two columns will no longer align - the property "all the cells of the same row have the same height" is broken.
Another example of an irreducible 2D grid layout is when you need both the horizontal and vertical scrollbar. Although it is likely just a convention, I noticed a previously overlooked detail when implementing this view myself: the two scrollbars actually don't overlap at the bottom right corner, instead, they only touch at a single point, and thus leaving a tiny square there. Now if I were a conformist, then I'd have to use "grid" instead of "pack" to achieve this arrangement, because overall it is actually a 2x2 table, even though the bottom right cell is so minute and contains nothing. (But I'm a rebel, so I insisted on using pack for consistency throughout my layout code.)
To recap, my opinion is, "pack" deserves to be better understood, promoted, and even recommended as the default, go-to option to implement the majority of UI layouts (or the majority of levels in a layout tree), before resorting to grid. Furthermore, "pack" could be introduced first to Tk learners before "grid". Although I have to admit, conceptually, grid is a much more familiar concept to most people, because we are so used to looking at tables in daily life. But once you read the "packing algorithm", it will become very intuitive, so that won't be too much of an obstacle.
Scrolling
Scrolling in Tk was a shock to me. Now I understand the situation a little better - we've just been spoiled by web browsers too much; they've nailed the perfect abstraction layer for enabling and configuring scrolling on any generic HTML element, and that's what I expected from Tk at first. Tk hasn't found such an API that abstracts away scrollbars - you have to create every scrollbar widget explicitly, and connect it to the corresponding content widget to be scrolled, in a bidirectional way. That's OK per se, what's much worse is that not many widgets support scrolling, in particular, the generic container widget, frame, cannot be scrolled, that was to be honest the single trigger of outrage throughout the Tk journey - I thought of giving up for a while, for such an API oversight was hard to imagine when we're so used to having scrollbars on divs for free with HTML and CSS.
But I soon cooled down and kept on learning, after all, Tcl/Tk had been nice and empowering up till this one hiccup, and it wouldn't be right to abort the journey altogether because of it. People must have needed to scroll an arbitrary widget in Tk! I did saw a few third-party implementations of scrollable "frames", but as a learning project, I insisted on staying with Tk proper. And I read that the standard way to enable scrolling on a frame is to place it on a canvas, which is one of the versatile GUI powerhouses in Tk where one can do very custom graphical crafts.
I didn't want to dive into the rich drawing API of the canvas widget, because again, no premature learning. What's relevant here is, to enable scrolling of our canvas (on which our actual frame is positioned), we must specify the "scroll region" as the area occupied by the frame, which can be obtained using the bbox (bounding box) subcommand of canvas. More importantly, this scroll region must be timely updated as the frame size changes (the frame contains a dynamic list of download tasks, each being also a frame). This is achieved by bind <Configure>, like an "onResize" event handler.
All the above is just standard Tk's way of implementing generic scrolling for some frame. Once you learn it, it's done. But I encountered a particularly curious problem with maintaining the auto-resize property (always filling the window width) of the frame that sits on the canvas. By far, my understanding of the issue is as follows:
Although canvas as a widget also acts as a geometry manager (other such dual-role widgets include panedwindow, text, etc), it lacks the auto-resize options offered by grid (with -sticky and -weight) and pack (with -fill and -expand). Therefore, it's impossible to set the frame to always fill the canvas width as the canvas changes its size according to the top-level window width (canvas can do so because it's managed by pack).
Now the most tricky part is, it doesn't work even if you explicitly set the frame width to that of the canvas, again using "bind", instead, it will still take minimal horizontal space, namely the width of the narrowest task item in the list. Why?
Because geometry managers, both grid and pack included, effectively shrink-wraps the widgets in the end, after finishing all the previous arrangement steps, minimizing the overall rectangle. The only way to break that force is by properly configuring the auto-resize property as mentioned previously. But we just can't do that for our frame sitting on a canvas, yet that's the only way to implement scrolling for it.
For completeness, this is how a canvas manages a frame (no way to configure auto-resize):
.f.c create window 0 0 -anchor nw \
-window .f.c.list
The solution I've found, a bit hacky but proven to work, is as follows:
Into the frame, insert a hardcoded "placeholder" as the first dummy, almost invisible item in the task list (a line of 1 px height). Now we use "bind" to set its width to always equal that of the canvas. Finally, we use "pack" to configure it to fill the width of the containing frame. This will initially hold the frame to be as wide as the canvas, like a "stent" preventing the space from collapsing, and subsequently ensure that it always auto-resizes horizontally according to the window width.
frame .f.c.list.placeholder
bind .f.c <Configure> {
.f.c.list.placeholder configure -width %w
}
pack .f.c.list.placeholder -fill x
Other Odds
Event propagation: unlike JavaScript's default behavior, in Tk, there is no automatic "bubbling", or event propagation up the widget tree; consequently, an event handler registered on a parent widget, for example, cannot be triggered by that event taking place on its children. This makes some common GUI interactivity implementation more verbose, but again, perhaps Tk had different perspectives than web browsers back then. On the other hand, I appreciate explicitness as well, so that's OK with me. The way to enable specific bubbling (e.g. propagating to a specific ancestor widget, where the event handler is registered) is via bindtags.
Variable binding: text-based widgets like label can be configured to two-way bind to some global variable via -textvariable. It's quite handy, but also limited compared to a fully "reactive" architecture, esp. like Elm, where the view is an arbitrary function of the model, where you can do all kinds of derivations before presenting some app data. By far I don't need heavy computations, mostly just displaying data verbatim, but when such needs come, I must figure something out - perhaps with Tk being imperative in nature means I just have to do the view updates with explicit code? But essentially I need a way to monitor for changes in the global variable from which the view is determined, right? trace seems to be relevant?
Starpack: very cool for a quick executable distribution! Not sure if SDX is capable of more advanced configurations for executable generation though. But I'm generally pretty clueless in that space, will look into that sort of stuff one day.