Your users don't want to wait
As we come into the New Year, one of Google's goals is to try and make the web a much faster place. One of the ways that they have talked about accomplishing this is by adding load times as a factor to the algorithm.
We've already seen Google's movement towards a faster web experience with Chrome and load times effecting Adwords landing page quality. Now it looks to be pushed into organic search. The weight placed on load times for organic search will probably be quite minimal compared to other top factors but this doesn't mean that it should be brushed aside.
In fact, I would recommend optimizing your pages for faster load times even if it wasn't going to be influencing Google's algorithm. From a user's standpoint the last thing you want is for pages to take time to load up. We live in a society where we want information, and we want it now. Remember the old days where we would wait 5 seconds for an image to load? Not anymore, for those that spend hours a day on the internet whether for work or just for fun, any load time waits can be really annoying. In fact it's annoying to the point where people will avoid any site that doesn't load quickly. Competition gets tougher all the time, so if there are other faster equivalent websites (which there always are) you may lose potential customers.
To help us optimize our sites, Google has given us some resources to take advantage of including some tools to use, the 'Site Performance' section within Google Webmaster Tools and articles on best practice.
Get you load time under control, not only for Google and its algorithm, but also for your users.