An accelerator manager manages multiple accelerators in multiple programmable devices. An accelerator cast out policy specifies criteria for casting out accelerators in the programmable devices. The accelerator manager monitors usage of the accelerators by one or more computer programs, and generates a historical log from the monitored usage. When the conditions in the historical log satisfy criteria in the accelerator cast out policy for casting out an accelerator, the accelerator manager casts out the accelerator, and updates a virtual function table to replace calls to the accelerator that was cast out with calls to the software library.