Can FreeRTOS run on a AT90S8535 without modification?
Or does it need a new port?

If its an AVR then take a look at the section of the FreeRTOS WEB site about using a part other than an at323. However if it just has 512 bytes of RAM then this is not really enough to do anything useful.

Its an AVR but can't get it running because it returns from the function vTaskStartScheduler()

vTaskStartScheduler() will return if there is not enough RAM to create the idle task.

Thanx, I will try to replace the MCU for an ATMEGA32 will have more RAM available and is pin compatible.