I can't believe I'm praising Tcl

April 21st, 2008

So the other day I both defined Tcl procedures all by myself and used them interactively, and I liked it, to the point where it felt like some kind of local optimum. This entry is an attempt to cope with the trauma of liking Tcl, by means of rationalizing it. I'll first tell the story of my fascinating adventures, and then I'll do the rationalizing (to skip the oh-so-personal first part, go straight for the bullet points).

Here's what I was doing. I have this board. The board has a chip on it. The chip has several processors in it. There's a processor in there that is a memory-mapped device, and I talk to it using CPU load/store commands. The CPU is itself a JTAG target, and I talk to it via a probe, issuing those load/store commands. To the probe I talk via USB using an ugly console that can't even handle the clipboard properly. The console speaks Tcl.

Net result: in order to talk to the memory-mapped processor, I have to speak Tcl. Or I can use a memory view window in a graphical debugger. Except some addresses will change the processor state when read (pop an item from a LIFO, that sort of thing). So no, memory view windows aren't a good idea – you have to aim for the specific address, not shoot at whole address ranges. Damn, just who thought of defining the hardware in such a stupid way? Why, it was me. Little did I know that I was thereby inflicting Tcl on myself.

Anyway, there's this bug I have to debug now, and as far as I know it could be in at least three different pieces of software or in two different pieces of hardware, and I don't like any of the 5 options very much, and I want to find out what it is already. So at the ugly probe console I type things like word 0x1f8cc000 (reads the processor status), word 0x1f8cc008 2 (halts the execution), word 0x1f8cc020 0x875 (places a breakpoint). I get sick and tired of this in about 2 minutes, when the command history is large enough to make the probability of hitting Enter at the wrong auto-completed command annoying. It's annoying to run the wrong command because if I ruin the processor state, it will take me minutes to reproduce that state, because the program reads input via JTAG, which is as slow as it gets.

So I figure this isn't the last bug I'm gonna deal with this way, and it's therefore time for Extending my Environment, equipping myself with the Right Tools for the Job, Tailored to my needs, utilizing the Scripting capabilities of the system. I hate that, I really do. Is there something more distressing than the development of development tools for the mass market of a single developer? Can a programmer have a weakness more pathetic than the tendency to solve easy generic meta-problems when the real, specific problems are too hard? Is there software more disgusting in nature than plug-ins and extensions for a butt-ugly base system? But you know what, I really fail to remember 0x1f8cwhat the breakpoint address is. This story has one hexadecimal value too much for my brain. OK, then. Tcl.

I decided to have one entry point procedure, pmem, that would get a memory-mapped processor id, 'cause there are many of them, and then call one of several functions with the right base address, so that pmem 0 pc would do the same as pmem_pc 0x1f8c0000. Well, in Tcl that's as simple as it gets. Tcl likes to generate and evaluate command strings. More generally, Tcl likes strings. In fact, it likes them more than anything else. In normal programming languages, things are variables and expressions by default. In Tcl, they are strings. abc isn't a variable reference – it's a string. $abc is the variable reference. a+b/c isn't an expression – it's a string. [expr $a+$b/$c] is the expression. Could you believe that? [expr $a+$b/$c]. Isn't that ridiculous?

In fact, that was one of my main applications for Tcl: ridiculing it. I remember reading the huge Tcl/Tk book by Brent Welch with my friend once. There was a power outage and it was past the time when the last UPS squeaked its last squeak. And the book was there 'cause the hardware guys use it for scripting their lovecraftian toolchain. We really did have fun. Tears went down my cheeks from laughter. Even people with the usual frightened/mean comments about those geeks who laugh their brains out over a Tcl book didn't spoil it. So, ridiculing Tcl, my #1 use for it. The other use is the occasional scripting of the hardware hackers' lovecraftian toolchain. Overall, I don't use Tcl very much.

The nice thing about Tcl is that it's still a dynamic language, and reasonably laconic at that, modulo quoting and escaping. So I enter the usual addictive edit/test cycle using tclsh < script. N minutes down the road (I really don't know what N was), I've finished my 2 screenfuls of Tcl and the fun starts. I actually start debugging the goddamn thing.

$ pmem 0 stat
$ pmem 0 bkpt 0 0xbff
$ pmem 0 bkpt 1 0xa57
$ pmem 0 cmd run
$ pmem 0 stat
$ pmem 0 pc
$ pmem 0 rstack
3 return addresses
addr 0: 0x0005
addr 1: 0x05a8
addr 2: 0x0766
$ pmem 0 cmd stp
$ pmem 0 pc


You have no idea just how happy this made me. Yeah, I know, I'm overreacting. I'll tell you what: debug various kinds of hardware malfunction for several months, and you'll be able to identify with the warped notion of value one gains through such process. On second thought, I don't know if I'd really recommend it. Remember how I told low-level programming was easy? It is, fundamentally, but there's this other angle from which it's quite a filthy endeavor. I promise to blog about it. I owe it to the people who keep telling me "so low-level is easy?" each time they listen to me swear heartily at a degenerate hardware setup where nothing works no matter what you try. I owe it to myself – me wants to reach a closure here. Why should I tolerate being regularly misquoted at the moments of my deepest professional catharsis?

Aaaanyway, in just N minutes, I bootstrapped myself something not unlike a retarded version of gdb, the way it would work if the symbol table of my program was stripped. But no matter – I have addr2line for that. And the nice thing about my retarded debugger front-end is that it looks like shell commands: blah blah blah. As opposed to blah("blah","blah"). And this, folks, is what I think Tcl, being a tool command language, gets right.

I come from the world of pop infix languages (C/Java/Python/Ruby/you name it). Tcl basically freaks me out with its two fundamental choices:

So basically, pop infix languages (and I use the term in the most non-judgmental, factual way), pop infix languages are optimized for programming (duh, they are programming languages). Programming is definitions. Define a variable and it will be easy to use it, and computing hairy expressions from variables is also easy. Tcl is optimized for usage. Most of the time, users give simple commands. Command names and literal parameters are easy. If you are a sophisticated user, and you want to do pmem 0 bkpt [expr [pmem 0 pc] + 1], go ahead and do it. A bit ugly, but on the other hand, simple commands are really, really simple.

And eventually, simple commands become all that matters for the user, because the sophisticated user grows personal shortcuts, which abstract away variables and expressions, so you end up with pmem 0 bkpt nextpc or something. Apparently, flat function calls with literal arguments is what interactive program usage is all about.

I'm not saying that I'm going to use Tcl as the extension language of my next self-made lovecraftian toolchain (I was thinking more along the lines of doing that one in D and using D as my scripting language, 'cause it compiles fast enough and it's apparently high-level enough). I haven't thought enough about this, but the grotesque escaping/quoting in Tcl still freaks me out; I don't want to program like that. All I'm saying is that I like the interactive part. Specifically:

Allow me to elaborate.

Short code vs short commands

Lots of people have noticed that keeping your code short is extremely important. More surprisingly, many people fail to notice this, probably because "1 line is better than 5" doesn't sound that convincing. OK, think about 100K lines vs 500K and you'll get the idea. Oh, there are also those dirty Perl/shell one-liners that make one doubt about this. I've known a Bastard Programmer that used 2K bash one-liners as his weapon of choice. OK then, so the actual rule must be "short code is good unless it's written by a bastard". But it's the same core idea.

So we have the Architect type, who loves lots of classes which delegate work to each other, and we have the Enlightened type, who wants to write and read less. And the Enlightened type can rant and rave all day how Python, or Ruby, or Lisp make it oh-so-easy to define data structure literals, or to factor out stuff using meta-programming, or some other thing an Architect just never gets. And I'm all with it.

And then we have interactive shells. And in Python it's doit("xx","yy"). And in Lisp it's (doit "xx" "yy"), or (doit :xx :yy), or (doit xx yy) if you make it a macro. And in Ruby it's doit :xx :yy, if you use symbols and omit parens. And that's about as good as you can get without using your own parser as in doit "xx yy", which can suck in the (more rare) case when you do need to evaluate expressions before passing parameters, and doesn't completely remove overhead. Also note how all these languages use (), which makes you press Shift, instead of [] which doesn't. Ruby and Perl let you omit (), but it costs in readability. And [] is unanimously reserved for less important stuff than function calls.

The whole point of short code is saving human bandwidth, which is the single thing in a computing environment that doesn't obey Moore's law and doesn't double once in 18 months. Now, which kind of bandwidth is the most valuable? I'll tell you which. It's the interactive command bandwidth. That's because (1) you interact a lot with your tools and (2) this interaction isn't what you're trying to do, it's how you're trying to do it, so when it isn't extremely easy it's distracting and extremely frustrating.

This is why an editor that doesn't have short keyboard shortcuts for frequently used commands is a stupid fucking piece of junk and should go down the toilet right now. This is why a Matlab vector – [1 2 3] – is much better than a Python list – [1,2,3] (ever noticed how the space bar is much easier to hit than a comma, you enlightened dynamic language devotee? Size does matter). And don't get me started about further wrapping the vector literal for Numeric Python.

The small overhead is tolerable, though sucky, when you program, because you write the piece of code once and while you're doing it, you're concentrating on the task and its specifics, like the language syntax. When you're interacting with a command shell though, it's a big deal. You're not writing a program – you're looking at files, or solving equations, or single-stepping a processor. I have a bug, I'm frigging anxious, I gotta GO GO GO as fast as I can to find out what it is already, and you think now is the time to type parens, commas and quotation marks?! Fuck you! By which I mean to say, short code is important, short commands are a must.

Which is why I never got to like IPython or IDLE. Perhaps Ruby could be better, because of omitting parens and all. Ruby seems to be less inflicted with the language lawyer pseudo-right-thing mindset. But the basic plain vanilla function-call-with-literal-args syntax still doesn't reach the purity of *sh or Tcl. Well, the shell is an insanely defective programming language, so it's not even an option for anything non-interactive. But Tcl gets way closer to a programming language. Which brings us to the next issue:

Ad-hoc scripting languages – the sub-Turing tar pit

Many debuggers have scripting languages. gdb has one, and Green Hills MULTI has one. Ad hoc command languages usually get the command-syntax-should-be-easy part right – it's command arg arg arg... They then get everything else wrong. That is, you usually don't have any or some of: data structures, loops, conditionals and user-defined functions, option for expression evaluation in all contexts, interface to the host OS, and all the stuff which basically would make the thing a programming language. Or you get all those things in a peculiar, defective form which you haven't seen anywhere else.

I wish people stopped doing that. I understand why many people do that very well – they don't know any language which isn't a 3rd generation one (presumably C++ or Java). They don't know how scripting works except on a theoretical level. They know how to build a big software system, with objects and relationships between objects and factories of objects and stuff. At the system/outside world boundary they're helpless though. Outside of the system our objects are gone. There's this cold, windy, cruel world with users and files and stuff. Gotta have an AbstractInputParser to guard the gates into our nice, warm, little system, um, actually it's "big", no, make it "huge" system.

These are the Architects who get mocked by the Enlightened dynamic language lovers. They normally dismiss scripting languages as "not serious", therefore, when faced with the need to create a command language for their system, they start out with a plan to create a non-serious (a.k.a crippled) language. Even if they wanted to make it a good one, they never thought about the considerations that go into making a good scripting language, nor do they realize how easy/beneficial it is to embed an existing one.

So basically we have 3GL people, who realize that commands should be short ("it's a simple thing we're doing here"), but they don't see that you need a real Turing-complete programming language for the complicated cases. And we have 4GL people, who optimize for the complicated case of programming ("what's a scripting language – it's a programming language, dammit!"), and they don't care about an extra paren or quotation mark.

And then we have Tcl, which makes easy things really easy and scales to handle complicated cases (well, almost, or so I think). And not only does it make plain funcalls easy – it reserves [] for nested funcalls, in the Lispy prefix form of outercall arg [innercall arg arg] arg... [] is better than (). Pressing Shift sucks. And custom keyboard mapping which makes it possible to type parens without pressing Shift is complete idiocy, because you won't be able to work with anyone's machine. This shit matters, if you program all day long it does.

Now what?

I don't know if I'd use Tcl. It's less of a programming language than your typical pop infix 4GL. For starters, [expr] is a bitch. And then there are "advanced" features, like closures, that I think Tcl lacks. It has its interesting side from a "linguistic" perspective though. It has really few core syntax, making it closer to Lisp and Forth than the above-mentioned pop infix ilk. So you can use Tcl and claim for aristocracy. Of course you'll only manage to annoy the best programmers this way; the mediocre won't know what you're talking about, seeing only that Tcl doesn't look enough like C to be worth the name of a language.

I'd think a lot before embedding Tcl as a scripting language for my tools, because of linguistic issues and marketing issues (you ought to give them something close enough to C, whether they're a customer or a roommate). So the practical takeaways for me are modest:

  1. I ain't gonna mock Tcl-scriptable tools no more. I understand what made the authors choose Tcl, and no, it's not just a bad habit. On a level, they chose probably the best thing available today in terms of ease-of-programming/ease-of-use trade-off (assuming they bundle an interactive command shell). If I need to automate something related to those tools, I'll delve into it more happily than previously. So much for emotional self-tuning.
  2. I'll let it sink, and try to figure out whether you have a better trade-off. For example, if Ruby had macros (functions which don't evaluate their inputs), you could say doit x y without making x and y symbol objects, which forces you to prefix them with a colon. How macros of that sort should work in an infix language escapes me (not that I think that much about it, but still). Anyway, I'll definitely add Tcl to the list of things I should understand better in order to fight my linguistic ignorance. Being an amateur compiler writer, that seems like one of my duties.

1. CGMApr 22, 2008

No, we don't have closures in Tcl – some discussion of this can be found at http://wiki.tcl.tk/3330 .
Yes, [expr] is a bit clunky – in Tcl8.5 arithmetic can also be done with prefix operators:
% namespace import ::tcl::mathop::*
% * 3 [+ 1 2]

2. Dossy ShiobaraApr 22, 2008


I'm so glad you've put aside language bigotry and evaluated Tcl fairly–when you do, it's easy to see how convenient it can be for some tasks.

Of course I'm biased, but I also think Tcl is a fantastic language for developing web applications–thus, my affinity for AOLserver.

When you reduce web development to the simple process of "consume bits from a data source, transform strings, output bits to a network socket" ... Tcl's simplicity really makes rapid development a breeze, coupled with AOLserver's library of Tcl procs to ease some common tasks.

I hope more folks give Tcl a fair shake, given it's one of the oldest and arguably the most mature scripting language out there.

3. Hello Script for Tcl » Bin-BlogApr 24, 2008

[...] I can’t believe I’m praising Tcl [...]

4. Weekly linkdump #133 - max - блог разработчиковSep 5, 2008

[...] хорош язык Tcl, I can't believe I'm praising Tcl. У меня был очень похожий опыт и воспоминания от тикль [...]

5. Bernard DevlinDec 10, 2008

Hey, thanks for this. I looked at tcl 10 years ago, and dismissed it for python and ruby (I can't even remember why now). Anyway, I've recently been wishing that my main GUI programming language (a modern version of hypertalk – http://www.runrev.com) was a) more extensible, b) had multi-threading support. This week I stumbled upon another article defending tcl, and on reading further I found out tcl was just about _the_ most extensible language, and had implemented threading in exactly the way I thought threading should be implemented. Then I started reading about the new OO core features coming in tcl 8.6, and they are look like the most appealing way of doing OO that I've ever seen.

6. Yossi KreininDec 10, 2008

Interesting, the stuff that you're doing. I've lost hope for a "natural programming language" long ago, and still, I'm curious to look at your thing sometime.

7. MichalMar 12, 2009

'[' is easier than '(' only on some layouts (such as the us one).
On the Swedish keyboard for example, shift is still needed for '(' but in order to get '[' you need right alt (a.k.a. alt gr).
Now, which one is better?

This points out that designing a syntax after a keyboard is a tricky business. Some national layouts make it clear that they are made with no consideration of programming needs what so ever.
I definitely think there should be several layouts tuned to particular needs (but similar enough to still be somewhat usable by different users).
Programmers use the national characters (such as åäö) less often than operators, so why are we forced to work on keyboards that have dedicated keys for (åäö) but squeeze up to 3 important operator-characters on other keys?
For your information, here are some examples (plain, shift, altgr):
2 " @
7 / {
8 ( [
9 ) ]
0 = }
+ ?
| -this sucks particularly if you work with a shell
¨ ^ ~ -all of them are normally "dead" meaning they get attached to other characters so you need to type them twice and then delete one to get a single one.

8. MichalMar 12, 2009

&lt &gt | was eaten by evil html there

9. Yossi KreininMar 12, 2009

I think that the problem of being a minority, linguistically, is a pretty big problem, which is illustrated by the fact that I'm typing this in an approximation to English and not in a language I actually know.

I think that if I lived in a country with programmer-unfriendly keyboard layouts, I'd keep a QWERTY keyboard at work and use a virtual keyboard for communicating with fellow human countrymen (actually for a variety of reasons I do it today, using translit.ru and mikledet.com instead of a trilingual keyboard, although English-Hebrew-Russian keyboards are commonplace in Israel.)

10. Chris EdwardsMay 13, 2009

That was a great read, thank you. I went through a similar process recently. I'd used Tcl a couple of years ago while dabbling with Expect (an essential utility in its own right), and mostly saw it as just another scripting language but with an eccentric range of different types of bracket. After spending some time over at http://wiki.tcl.tk/, I started to realise how easy Tcl is to misjudge. There's some beautifully succinct code over there. And Tk is a breath of fresh air for rapid GUI development.

Tcl is what I imagine someone would come up with if they set about re-designing your typical unix shell language from scratch: trying to do it properly, cleanly, extensibly. The seemingly familiar syntax disguises the language's most interesting features, such as everything being a string, and all the control structures actually being commands. I sometimes find myself thinking that Tcl is more akin to Lisp than it is to C, and what seemed like idiosyncrasies become part of the beauty of the language: the simplicity and consistency of it. But, yeah, I agree that [expr] is pretty hard to like, and the "if 0 { ... }" block commenting technique, with its balanced-braces requirement... :^)

Yossi, are you saying you're not fluent in English? Your written English is better than most I've seen on the Web!

11. Yossi KreininMay 13, 2009

Regarding Tcl – most people don't like (+ a b) any more than [expr $a+$b]...

Regarding English – thanks! I think I'm alright for a foreigner, but there's the occasional error I make, and there's the worse problem of having to think lots while writing in order to not make the other errors.

12. gus3Oct 13, 2009

"You have no idea just how happy this made me."

Ha! Yes, I do. I've given myself an assignment to touch various languages (common and esoteric), using the Collatz function as a basis, and using both cores of my AMD64X2 if possible. My most recent language tackled was Python. Imagine my surprise on finding that Python had already implemented multi-core iteration:

from multiprocessing import Pool
def iterated_function(x):
blah blah blah python code here...

print pool.map(iterated_function, range(1,n))

I was doing a happy dance when it worked just the way I thought it would. I hope my downstairs neighbor wasn't too annoyed with me.

13. Yossi KreininOct 14, 2009

Never looked into Python's MP support. Looks like TBB or TPL.

14. Peter CordesNov 21, 2009

I'm lucky (in some ways) that I'm a native English speaker, so the worst keyboard annoyance for me is tall enter keys that make the pipe and backslash key harder to hit. (I'm a GNU/Linux command line junkie, so I pipe all the time.) Although I always map the key labelled CapsLock to be another Control key. Here in Canada, some laptops come with unfortunate Canadian-bilingual (french) layouts. I recently found a laptop model that I would have bought except for the keyboard. :/

As for alternate keyboards for multilingual input: if you use it often, maybe it would be convenient to have two physical keyboards? I often grab the kbd and hold it in my lap while I lean back in my chair, so it's probably less convenient or would require more desk space and time for those who like their keyboard on a fixed surface.

15. Yossi KreininNov 21, 2009

Actually you can easily buy a trilingual English/Hebrew/Russian keyboard in Israel, however not being able to count on having one at a given workstation got me hooked on transliteration, so I can't type in Russian using a Russian keyboard layout anywhere near the speed of my typing using an English layout and the transliteration convention of http://translit.ru. At least it works everywhere as opposed to any combination of physical keyboards one can set up at a particular workstation.

16. TomSep 18, 2010

Your Ruby without symbols. But I must say it freaks me out a little ;)

def method_missing sym, *args
return sym

puts whatewer

17. zokierJan 1, 2011

imho swedish kb layout is just horrible. programmers across scandinavia would be better off with a new, us-intl -based, layout. i myself as a finn am using custom layout, which is kinda annoying when using other computers.

but back to the topic, tcl looks interesting. i do not particulally like bash, but using 'real' languages repl as shell feels kinda clunky. would tcl work as a login shell in your opinion?

18. ChrisJan 1, 2011

Another great thing about tcl for the c programmer is that adding new commands to tcl is dead simple, exactly like a cs1 programming class, you get argv and argv and build up your stdout, I mean TclResult.

Your new programmers can be given the job to extend your tcl knobs for tweaking your big system's behavior.

19. Yossi KreininJan 1, 2011

@zokier: regarding tcl as a login shell – an interesting idea and it might very well work, however I'd guess you need to tweak it, starting with having it run processes whenever there's no command defined, and then completion and stuff – tclsh probably wouldn't work. So it's the question of whether anyone did the tweaking and made it available, similarly to the way the did for Python with pysh and similar (not that I'd recommend that), or for Scheme with scsh, etc.

20. mahtJan 2, 2011

5 easy letters

21. Yossi KreininJan 2, 2011

@maht: dumps core. Discussed here though.

22. Jouni OsmalaJan 2, 2011

Here are the only special characters I can get with single key press without going to numpad ,.-'+§<

It sucks to code with Finnish keyboard layout. Swedes probably use same. 3 extra vowels make it hard. Hmm.
Hmmm. This discussion makes me think that should I configure full custom keyboard layout for coding.
If getting rid of that shift is such a great advantage in programming.

23. Yossi KreininJan 3, 2011

Well, some think that it doesn't really matter that much, that shift and all, if you're a good typist. I'm a pretty lousy one, I guess.

24. nightwatchJun 16, 2012

Few years ago I was interfacing a motion control processor (for controlling servos in CNC machines) and got tired of C compilation. As a side tool I created a Tcl wrapper for all motion chip API and then the real fun began. It was astonishing how much easier everything became. Tcl is a powerful language that makes things easy , especially the things that are ridicously hard in C (meta-programming, expression eval, dynamic data structures, threading, network communication and many more). I only wish that Tcl was easier to embed in other languages, not only C but C# and Java too – it could become a perfect application scripting library.

25. nightwatchJun 16, 2012

And, @yossi and @zokier, I have used the tcl interpreter as a linux shell. It was a continuation of above mentioned CNC project – a micro linux distribution with only necessary tools, uclibc, busybox and tclsh shell (12 Mb total). All the CNC api was exposed as shell commands so you could just log on and start issuing robot control commands. Unfortunately, the product was never really finished...

26. Richard HollerithJun 16, 2012

It's no big deal, but "rationalize" probably does not mean what you think it means.

27. Mikhai KorobovJun 16, 2012

By the way, there is %autocall directive in IPython:

In [1]: %autocall
Automatic calling is: Smart

In [2]: def foo(x): return x*2

In [3]: foo 5
——> foo(5)
Out[3]: 10

28. Ousman BereztJun 26, 2012

As a curiosity:

The only characters that are available unshifted on (practically) all keyboards are letters a-z, digits 0-9 punctuation ,. and addition/subtraction +-

For all other characters, there is a very large percentage of keyboards where some form of shifting is necessary.

All the paragraphs that mention () being vastly superior to [] are therefore factually incorrect.

So yeah, that's silly :-)

29. AFCDec 28, 2012

I've been inspired to change my keymap so that I don't have to press shift for ().

30. hmijailSep 12, 2018

I thought I'd mention that macOS offers 2 keyboard layouts for Polish language: "Polish" and "Polish Pro", where "Pro" seems to mean "programmer" and imitates an US keyboard in the location of symbols.

Back to Tcl... interesting how it keeps popping up here and there, directly and indirectly. Same as Ousterhout himself, by the way. Mhm...

Post a comment