Juliaca has flights to Lima for USD59-89 and to Arequipa for USD49-69. If you fly directly, take it easy the first few days, as the height will slow you down. You might suffer symptoms of altitude sickness.
There are busses to Cuzco, 7 hours and 9 USD, Arequipa, USD 11, 11 hours and Lima, up to 40 hours. You can get a direct bus to La Paz, but you might consider breaking the trip at Cococabana or even take a tour that visits some of the islands along the way.
There are also trains to Cuzco, which are rather expensive, because of the special tourist prices and no longer faster. They are also infrequent, leaving only every other day or so.