By default, the display order in the  [novelist-books] shortcode (and the automated /books/ archive) is “menu_order”. That means your books will appear in the same order they’re listed in Books > All Books.

A quick and easy way to change the order is to install a plugin called  Simple Page Ordering. Once it’s activated, return to Books > All Books. Now you can hover over your book title (watch the cursor change), then click, drag, and drop to reorder the books however you want.