ALSA is the national bus company although from madrid u might be able to find some other companies that are running buses to san sebastian for fairly cheap. RENFE the national train company is efficient and reliable. the new trains that speed around spain dont extend so much into San Sebastian, they basically get you from madrid to seville or cordoba and to barcelona, although the trains to SS are certainly suffient.
as far as the cheapest mode of transport your probably looking at buses. however, train travel in spain is usually cheaper than alot of the rest of western europe so u might not find it unreasonable to travel by train.
as far as booking ahead. you shouldnt have to, particularly from madrid to SS, there will no doubt be several trains and probably a few buses running that route everyday.
check out the train website renfe.es and the alsa.es website. you could also just do a generic google search for how to get between the two cities. you might come up with a bus company that operates between them that is cheaper, sometimes u have to look for those though.
as far as transportation to/from the airport. if your flying into madrids main airport, transportation to/from the bus/train station is cake. the metro runs right to the airport and you can get anywhere for like a euro.
you may want to investigate a little bit though on the web b/c madrid has two train stations Atocha and Charmartin. both of them operate service to different places and sometimes to the same places but prices and times may be different. for example, the train from Atocha to Barcelona is like 3 or 4 hours and costs 100euro, the one from Charmartin is like 8 or 9 hours overnight and costs like 40 euro.
so check that out as well.
its not as confusing as it all sounds, and if your totally lost, just find your way to a computer and travelpunk. someone always seems to be an endless bucket of knowledge about even the most random of places.
the world is a book, and those who don't travel only ever seen one page