R is an environment for statistical data analysis and graphics. MATLAB's origins are in numerical computation. The basic language implementations have many features in common if you use them for for data manipulation (e.g., matrix/vector operations).
R has statistical functionality hard to find elsewhere (>2000 Packages on CRAN), and lots of statisticians use it. On the other hand, MATLAB has lots of (expensive) toolboxes for engineering applications like
- image processing/image acquisition,
- filter design,
- fuzzy logic/fuzzy control,
- partial differential equations,
- etc.