You can — and most definitely should — because it’s a source of powerful, ranged magic attacks. If you want, you can wield a Sword and Grimoire and cast the spells from the book directly.