Equally important to the numeric permissions is the file ownership. For example, ownership of apache:www with permissions of 750 means that the apache user can read, write and execute; anyone with the www group can read or execute; and the file is protected against access by anyone else. Note that “execute” means two entirely different things for directories than for files!
In general, you want your permissions set such that your webserver can read and write (recursively) to config.inc.php’s files_dir, and to ./cache/, and ./public/. Optionally, for added features and reduced security, you can enable read and write to ./plugins/ and perhaps to the locale .xml files.
Your webserver should have read-only access to all other files and directories distributed in the package.