Matlab on OS X uses X11 to generate plots. These plots often have jagged lines, and are not rendered as smoothly as a PDF exported version of the same plot. With Octave, one can use the Aquaterm plotting front-end, but there seems to be no way to do that with Matlab. The following example piece of Matlab code makes Matlab convert a figure to PDF and display it in TeXShop, which auto-refreshes the PDF (unlike Preview):
figure(1);
set(gcf, 'Visible', 'off');
plot(x, y);
print(gcf, '-dpdf', 'figure1.pdf');
system('open -a Texshop figure1.pdf');
[robg adds: I can't test this one; if you can verify that it works, please post a comment.]
Mac OS X Hints
http://hints.macworld.com/article.php?story=20050401171844461