I'm looking into selling on Amazon (not FBA, storing and sending the items myself) and from what I understand (I could be wrong), the fees look higher than Ebay (you can either pay 75p per item and a percentage of the final selling price in addition, or you can pay a monthly fee of £30 a month, then you don't pay the 75p per item, only the final percentage fee.
The percentage fee for books is 15.3%! Also there's a "closing fee" for books of 50p, increasing to £1.00 in January 2021...
Here's a link. I'm hoping I'm reading it wrong but don't think I am...
Amazon Fees
I would imagine the FBA fees are higher because you pay for storage too (although their delivery charges will probably be cheaper).
Based on the above being correct I would say Ebay would be best, but unless the books are really rare, do a Buy It Now, not an auction (or start the auction price at the price you want for it, then it's a bonus if it goes higher). There doesn't seem to be much interest in auctions anymore unless you have something really rare with little to no competition. Look at the completed listings - Buy It Nows seem to fetch considerably more than auctions for the same product whenever I look.