One of the things that can slow down your site are database queries at load time. As a general rule of thumb there should not be more than 25 queries to load your blog’s pages.
A very useful plugin to check the queries being made at load time is Debug Queries. If you are logged in to the site as an administrator, it will display a list of database queries being made at the end of your pages, helping you analyse all those queries and the plugins generating them. You will probably be surprised at how many queries some low-quality plugins are generating (and thus slowing down your site).