Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Install a new tcsh shell for better UTF8 support UNIX
I was (until very recently) frustrated with the seeming inability of OS X to live up to part of its 'Unicode supporting' hype. That is, if you create files or folders whose pathnames contain multibyte characters (e.g., japanese) in the Finder, it's pretty difficult to actually do anything with them via the command line (i.e., in Terminal.app). Of course, you should see the names of the files displayed properly if you have the terminal window set to display utf8, but some other important functionality is sorely lacking for files and folders whose pathnames contain multibyte characters:
  1. Dragging an icon onto a Terminal window will show "garbage" for any multibyte characters in the file or folder's pathname, making this otherwise wonderful convenience useless (it's nice to be able to type "cd ", and then drag a folder's icon from the Finder and hit return, but it only works if the folder's path doesn't contain any multibyte characters).

  2. You can't directly access such a file or folder in the shell by just switching to the proper input method (e.g., via Command-space) and typing the pathname. This also means that it's pretty hard to create a directory or file with multibyte characters in its pathname interactively (in the shell).

  3. If you have a custom shell prompt that shows your current working directory, and you happen to be in a directory with multibyte characters in its pathname, your prompt will be messed up.
I fought with this in various ways and experimented with various non-solutions until I realized that there was probably a newer version of tcsh (the default shell for OS X) that might have utf8 support (the one that ships with 10.2 only seems to deal with euc & sjis). At that point, I had resigned myself to having to build my own tcsh, but when i went looking for the source, I ran across a binary version of 6.12.00 for OS X. Now, thanks to Kazufumi Tomori's efforts, we can all have decent utf8 support in Terminal.app without waiting for 10.3 or whenever Apple gets around to it.

Note that I certainly don't claim any credit whatsoever for his work. However, I thought that perhaps those unable to read Japanese, but who are nonetheless interested in a shell with utf8 support, might benefit from my rudimentary translation and paraphrasing of his installation instructions. The DMG file can be downloaded from his site by clicking the "DL / 0.6MB" button under "tcsh-6.12.00 for v10.2 Jaguar," and it contains two "package" files:
  • tcsh-6.12.00-replace.pkg
  • tcsh-6.12.00.pkg
Basically, the only difference in the two packages is that "tcsh-6.12.00-replace.pkg" will overwrite the stock tcsh in /bin/tcsh with the new version, while "tcsh-6.12.00.pkg" will place the new version in /usr/local/bin/tcsh instead. Note that if you already have a file /usr/local/bin/tcsh, it would be overwritten by installing the latter package.

There are a number of different ways one could choose to install either (you only need one) of these packages. The one described in Tomori's documentation suggests that, if you're installing the "replace" package, you make a backup copy of your original /bin/tcsh first. That's a good idea. He also notes that if you use the other package to install the new version into /usr/local/bin, you will need to edit /etc/shells to include the newly installed program in the list of "approved" shells before you can actually use it.

As this hint has already grown excessively verbose, I won't discuss here what /etc/shells is, or how to use sudo or whatever to edit it, nor how to change your default shell, etc. However, I will tell you the approach I took, which should work fine for most folks ... note that in the following, (root)% represents my shell prompt, and I am logged in (as you might suspect) as root. If you're not up for messing with root, you could likely just as well prefix the mv and ln commands below with sudo.

After installing "tcsh-6.12.00.pkg," open a Terminal window and type:
(root)% cd /bin
(root)% mv tcsh tcsh.6.10.00
(root)% ln -s /usr/local/bin/tcsh .
Don't forget that last dot! Now, if you do ls -l tc*, you should see something like:
lrwxr-xr-x  1 root  wheel      19 Jun 10 17:02 tcsh -> /usr/local/bin/tcsh
-r-xr-xr-x  1 root  wheel  315136 Apr 11 17:27 tcsh.6.10.00
One last step: cd back to your home directory and edit (as you, not root) your .cshrc or .tcshrc file to add the following line(s):
  set dspmbyte = "utf8"
  #the following is optional (see below)
  set prompt = "%m(%n):%~%# "
Now, if you open a new terminal window and type set and hit return, you should see, among other things, a couple lines like the following:
  dspmbyte        utf8
  version tcsh 6.12.00 (Astron) 2002-07-23 (powerpc-apple-darwin)
    options 8b,nls [ ... ]
And you should now be able to do all kinds of utf8 stuff right from the shell, including using alternate input methods to create file and folder names in alternate scripts (admittedly, I haven't tried this for anything other than Japanese and some arbitrary 8-bit roman stuff and symbols, but it should work fine).

If you added the optional line to the .cshrc (or .tcshrc), your shell prompt will work right, too. I'd provide an example here, but since the macosxhints page ends up encoded as iso-8859-1, it wouldn't show up properly.

Finally, sorry about the length of this post; I tried to strike a balance between too much detail for experts, and not enough for newbies, but probably failed on both accounts.
    •    
  • Currently 2.50 / 5
  • 1
  • 2
  • 3
  • 4
  • 5
  (2 votes cast)
 
[8,827 views]  

Install a new tcsh shell for better UTF8 support | 16 comments | Create New Account
Click here to return to the 'Install a new tcsh shell for better UTF8 support' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Install a new tcsh shell for better UTF8 support
Authored by: deleted_user18 on Jun 13, '03 12:27:49PM

You can also install tcsh 6.12-1 using fink.

How can I find out the version of tcsh my system has installed?



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: deleted_user18 on Jun 13, '03 12:36:23PM

OK, I found the answer myself reading your hint more carefully. Shame on me... :-)



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: eno on Jun 13, '03 02:24:29PM

Doesn't work for me... :-(

For example, when I drag folders into the Terminal with Spanish characters I get the accents stripped off them (so they don't work).

I can't type them either, like I would in any other app.

And I've tried changed the "Window settings" in Terminal.app too, but they have no visible effect....



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: carsten on Jun 13, '03 02:47:50PM

I had to also add "-N --show-control-chars" to ls to get non-ascii chars to display in directory listings within the terminal (using gnu ls from fink).

Also unable to drag files with odd characters in the filename into a Terminal here too, the non-ascii characters are still stripped. :(



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: Ezekiel on Jun 13, '03 10:55:42PM

Great hint! Thanks alot. The information in itself wasn't all that valuable to me, I just used Fink instead, but the idea was something I had never thought of. Now all my scandinavian characters show up nicely in the terminal. Flawless and very, VERY appriciated :-D



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: deleted_user18 on Jun 14, '03 03:08:47AM

One time it worked (new tcsh and UTF-8 support in shell". I could display folders with äöü and software like dselect and nano also displayed umlauts correct.

Now the next day I again can't use dselect and nano this way. If I switch back to ISO Latin1 they work but folders with umlauts no longer work.

This very frustrating. And I agree everything Apple says about OS X and Unicode is just a big hype.



[ Reply to This | # ]
What about zsh?
Authored by: sapporo on Jun 14, '03 06:15:49AM

I'm running zsh 4.0.6-1 via fink, and I want proper UTF-8 support too! Anyone know how?

TIA,
-sapporo.



[ Reply to This | # ]
What about zsh?
Authored by: santin on Jun 15, '03 06:36:14PM

Add the following to your .zshrc

setopt printeightbit

It's not perfect, but it helps a bit...

Also, you should use the following flags for ls

--show-control-chars (for GNU ls)
-v (for BSD ls)


[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: jpf on Jun 14, '03 09:47:54AM
eno wrote:
Doesn't work for me... :-( For example, when I drag folders into the Terminal with Spanish characters I get the accents stripped off them (so they don't work). I can't type them either, like I would in any other app. And I've tried changed the "Window settings" in Terminal.app too, but they have no visible effect....
and then carsten wrote:
I had to also add "-N --show-control-chars" to ls to get non-ascii chars to display in directory listings within the terminal (using gnu ls from fink). Also unable to drag files with odd characters in the filename into a Terminal here too, the non-ascii characters are still stripped. :(
another thing to check is to make sure that in terminal, the encoding (found under the "display" entry (not "window", btw) in the "terminal inspector" (cmd-i) for a given window) is set to utf8 explicitly. also, don't forget that not only do all your changes (installing the new version, editing your .cshrc, etc.) need to be in place before you can try them, but that they will affect only new terminal windows opened after you've got everything set up. you might want to double check the version of tcsh you're getting when you spawn a new shell ... it should be 6.12.00 ...

as far as accented characters, since i use a plain us/ascii keyboard, my tests mostly involved double-clicking them in the character palette, from whence they seemed to show up normally at the cursor position in the terminal window, as they should with any app. also, now i recall that at least umlaut worked as well via an option key sequence (e.g., typing opt-u then o should give you an o with an umlaut over it (so at least we can write motörhead properly!))

there may be an issue there, however, with the font you're using for your terminal window ... you can check in the character palette to see if a given font has a glyph for a given character. i did notice some anomalies with the font i like to use, "lucida sans typewriter oblique": there are certain accented characters (e.g., one that looks like a z with a dot over it, unicode 0x017C) that seem to work ok when entered, but then don't show up properly in a filename via "ls", etc. i think this is because the font i'm using does not actually contain that character. interesting how osx does a fair job of substituting the missing double-byte characters (no kanji in lucida sans typewriter oblique either!), but kind of breaks a little for something that might be considered in some cases just high ascii.

also, regarding the version of ls, i can't really address that, because i'm just using the stock ls shipped with osx, which works fine for me. and i can't speak for those using the tcsh available via fink, since that's not the one referred to in the hint. it's possible that the two binaries were compiled with entirely different options, etc., which may explain the difficulty some folks seem to be having as well.

[ Reply to This | # ]

try using the -v switch with ls
Authored by: gabidanon on Jun 14, '03 11:34:51AM

To see Hebrew in file listings, I had to use "ls -v"; otherwise I see question marks.



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: carsten on Jun 15, '03 06:21:59PM

Found the problem with my setup.

For Finder drag-and-drop into the Terminal to work, I had to turn on the Escape 8-bit chars with Ctrl-V checkbox in the Terminal emulation settings.

[ Reply to This | # ]

Install a new tcsh shell for better UTF8 support
Authored by: beepotato on Jun 16, '03 10:24:52AM

Thanks ! It was my problem too.

In fact, I found that when you check this box and set the encoding to UTF8, drag-and-drop from the Finder to the Terminal and typing pathnames containing multibyte characters both work perfectly with the version of tcsh included in 10.2 (version 6.10.00). I tried it with french accented characters and chinese characters, both worked.

But it seems to be able to display multibyte characters only as backslashed Unicode numbers, so you might want to install version 6.12.00 anyway if you want these characters to display nicely. Still, version 6.10.00 is fully functionnal if you don't care about that.



[ Reply to This | # ]
Bash already is UTF8
Authored by: newbish on Jun 18, '03 09:04:37PM

Trying this out, I inadvertantly discovered that bash is already UTF8 in 10.2.6. In fact, better than the TCSH I installed via Fink.

I'll try out the package pointed out by JPF next to see how well it handles things.



[ Reply to This | # ]
Put tcsh away, use bash!
Authored by: digdog on Jun 18, '03 10:55:29PM
use bash, and add .inputrc in your home, which contained:

set convert-meta off
set input-meta on
set output-meta on

Restart Terminal. Done, then you can use any UTF-8 characters you want.

[ Reply to This | # ]
Put tcsh away, use bash!
Authored by: Spartacus on Oct 28, '03 08:57:55AM

Why should we use bash? Just because you say so? Is it so 'superior' that everyone should switch to it without thinking?

On Mac OS X 10.1, tcsh came with a default config that made it a lot more powerful and convenient to use than any other shell I have ever seen (in its default configuration) on any platform. Starting with Jaguar, this powerful config was disabled by default for new users but still available in /usr/share/tcsh/examples/.

I have not seen anything of the like for bash in Mac OS X. By default, aliases and completions are second to none.



[ Reply to This | # ]
Install a new tcsh shell for better UTF8 support
Authored by: Spartacus on Oct 28, '03 09:34:45AM
I noticed that following this hint to re-establish ~/Library/init/tcsh functionality in Jaguar and Panther breaks the effect of the "set dspmbyte=utf8" setting. It turns out that /usr/share/tcsh/examples/tcsh.defaults sets the nokanji shell variable. To undo this, add this line before set dspmbyte=utf8:
unset nokanji
nokanji is described in the tcsh manpage:
nokanji (+)
        If  set  and  the  shell  supports Kanji (see the version shell
        variable), it is disabled so that the meta key can be used.
I don't know what the meta key is but I know I didn't check "Use option key as meta key" in the Terminal's Window Settings.

[ Reply to This | # ]