Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Don't accidentally open huge text files from the Command Line UNIX
In my line of work, I frequently deal with very very large text files, gigs in size. Every once in awhile, I inadvertently open such a file using the open -e command from the command line to view it in TextEdit (without checking the file size first).

Whenever I do this, my system crawls to a halt as TextEdit's virtual memory size bloats and the swap space goes crazy with page swaps. It takes minutes for me to go around saving open files before I can force-quit TextEdit. Here's a way I've found that can avoid these headache-inducing periods of no work getting done.

This hint, unfortunately, only works if you happen to (like me) use the C shell as your shell of preference, because there's no way of referencing arguments in a bash alias. Perhaps someone can post a bash equivalent in the comments. Also, there are many valid reasons to avoid aliasing actual commands (though I do so all the time without any problem) so you may want to change the alias name to something other than 'open.'

Finally, I used perl because I'm most familiar with it, though I'll admit there may be a more efficient and simple way of doing it. Again, I'll defer to improvements in the comments.

Basically, all I did was create an alias for the open command which first checks the file sizes before performing the command. If one or more of the files supplied to open (with -e as the first argument) is larger than 200M, it prints a warning instead of opening the file(s). Thus, to get around the warning, you would have to escape the alias (e.g. \open -e myfiles). It's a nice check to make sure I'm not opening a very huge file.

Here it is:
alias open 'perl -e "@x=split(/\s+/,qq(\!*));(@x)[0] eq qq(-e) ? (scalar(grep{@y=(split(/\n/));(stat((@x)[(@y)[0]]))[7]>200000000}(1..(scalar(@x)-1)))?print STDERR (qq(WARNING: One or more of your files is awfully big to open in text edit: ),(stat(qq((@x)[1])))[7],qq(\n)):exec(qq(open \!*))) : exec(qq(open \!*))"'
Just put the alias in your .cshrc file and you're good to go. Note I went to great lengths to avoid using the dollar symbol due to shell interpolation. I'd be very interested in learning better ways to avoid the interpolation.

[crarko adds: I haven't tested this one.]
    •    
  • Currently 3.00 / 5
  You rated: 3 / 5 (5 votes cast)
 
[4,963 views]  

Don't accidentally open huge text files from the Command Line | 10 comments | Create New Account
Click here to return to the 'Don't accidentally open huge text files from the Command Line' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Don't accidentally open huge text files from the Command Line
Authored by: DaphneDiane on May 17, '11 08:17:03AM
From my experience bash actually has better argument handling then csh. Here's the same thing in bash. I called it edit so as to not overwrite the open command since that is also used to open other things besides text files.
edit() { for i in "$@"; do if [[ -e "$i" && ( $(stat -f%z "$i") -lt 200000000 ) ]]; then open -e "$i"; else echo "WARNING: $i is awfully big to open in a text editor" 1>&2; fi; done; }


[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: robleach on Aug 21, '12 11:03:29AM

Note, the code only manipulates the command if -e is supplied, so you can open other file types without worry. Besides, you can escape the command using "\open ...". And as far as argument argument handling, I have not figured out a way in bash to refer to arguments to reproduce tcsh's ability using "!*" and "!!:3-5", etc.. It could just be that I'm not familiar enough with bash to know how to do it.



[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: jaguarcy on May 17, '11 08:18:05AM
you could very easily put that in a file called 'open', make it executable, and put it somewhere like ~/bin/, and put ~/bin/ first in your $PATH. then it'll work irrespective of shell :)

#!/bin/csh
...the above code here, minus the alias open '...' part...


[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: adrianm on May 17, '11 08:18:54AM
With bash (the default shell in OSX) you could use a function rather than an alias, eg:

function open {
  echo $1 $2
}
You can probably adapt the hint to use that, and it would probably be more readable too.
---
~/.sig: not found


[ Reply to This | # ]
bash function
Authored by: lullabud on May 17, '11 09:00:06AM
Here's a bash function to do pretty much the same:
function open {
size_limit=10000000000
biggest=$(stat -f%z "$@" | sort -n | tail -n 1)
if [ "$biggest" -gt "$size_limit" ] ; then
   echo "One of the files is too big" 1>&2
else
   /usr/bin/open "$@"
fi
}
Edited on May 17, '11 09:08:29AM by lullabud


[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: CarlRJ on May 17, '11 12:06:45PM
Ugh, TextEdit, really? For large (text) data files, do yourself a favor and use a tool more appropriate for the job, like less or vi (which is really vim on Mac OS X). The fully GUIfied MacVim is the best of the old and new worlds for working with text files of any size.

[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: robleach on May 23, '11 09:42:15AM

You know, there are plusses and minuses to every app. I use vim somewhat frequently, but there are lots of things you can do in Text Edit that you cannot do (or cannot do easily) in vim or some other text editors. In fact, I prefer xemacs exclusively for coding. But some nice features in Text Edit include for example, option-click-drag to make vertical selections, which I frequently use for tab-delimited data. I also perform multi-line searches very often. There are other good reasons to use Text Edit which I won't go into. I'll agree that vim is pretty awesome and can likely perform functions like these in its own way (given some effort), as probably could other text editors, but Text Edit isn't as bad as you think. Some people write off an app because it doesn't have feature 'X'. If that's important to what you do, that's cool, but you could be missing out on features you never knew existed that you might find very useful - or perhaps you're one of a few that like feature 'X' and it's not useful for other people. It depends on your specific use-case. But you're definitely right about other text editors being better with large files. In my case, I'd rather just `more -S` a large file and if I need to edit it, I'll use vim. But typically, I just want the file to appear in a different window next to my terminal. Do you know how to do that with a single terminal command? *That* would be cool.

Rob



[ Reply to This | # ]
Don't accidentally open huge text files from the Command Line
Authored by: CarlRJ on May 23, '11 12:57:53PM
[ I'm continuing this not to try to say "thou shalt use MacVim above all others", merely following up on your points. ]
But some nice features in Text Edit include for example, option-click-drag to make vertical selections, which I frequently use for tab-delimited data.
FWIW, Option-click-drag for column selections works just fine in MacVim
I also perform multi-line searches very often.
Not sure precisely what this means; Vim (GUI or otherwise) can perform any search you can possibly imagine. This may help: Search across multiple lines - Vim Tips Wiki
In my case, I'd rather just `more -S` a large file and if I need to edit it, I'll use vim.

I normally do the same, though converted from typing more to less back in the days when they were different programs.

I totally get that some people just prefer different apps (spending most of my time in Terminal means non-typical Mac user right off the bat), I was trying to make the point that, for large column-oriented data files, Apple's TextEdit is (surprisingly, given the name), often not the best tool for the job.

But typically, I just want the file to appear in a different window next to my terminal. Do you know how to do that with a single terminal command? *That* would be cool.

Install the mvim command-line component that comes with MacVim. Type "mvim filename", or more excitingly, "some command | mvim -". Results pop up in a separate window, which you can dismiss at your leisure with ":q!", Cmd-W, or clicking the red X gumdrop (I normally use Cmd-W, Cmd-D, to quickly close, dismissing the "Do you want to save" prompt).

These days, I find that a lot of things I used to do as "some command > /tmp/foo" followed by "vi /tmp/foo" have been simplified/improved to "some command | mvim -".

Note that TextWrangler, BBEdit, TextMate, SubEthaEdit, and likely others, all supply some similar command-line executable to open a file in them, you're not stuck with MacVim for this :)



[ Reply to This | # ]
Use a better text editor
Authored by: hayne on May 17, '11 01:08:56PM

I agree with CarlRJ - the correct solution would be to use a better text editor. E.g. TextWrangler can deal with very large files with no problem.
TextEdit is a good choice for a (simple) word-processor, but not so good for a text-editor (for plain text files).



[ Reply to This | # ]
Use a better text editor
Authored by: asmeurer on Aug 12, '11 05:12:01AM

Maybe it's improved in recent years, but I once tried opening a large file in TextWrangler, and it just hung there, assumedly trying to load the whole thing into memory. I ended up learning to use emacs enough just to read it, which worked fine. I'm sure less or vim would work just as well.

One could argue that these command line tools have a much better legacy and hence are more stable with respect to these things. Or perhaps the fact that they only have to render one terminal screen at a time gives them a better loading model (it's hard for a GUI editor to dynamically load different parts of a file into memory when they all live in the same scroll area).



[ Reply to This | # ]