When can I go to the left side of Liberty City in GTA 4?

There's a mission called Four Leaf Clover, which involves a bank heist. Once you've done that, you'll have full access to the whole of Liberty City.