One reason could be the filesize: php can only upload up until a certain (php.ini) filesize (mostly about 2MB) It doen't know the filesize in advance, so upload will start, but then just not be completed. Follow a similare discussion here:
Next: SiverStripe has predefined a hughe number of file extensions as acceptable for upload, but djvu is not part of them. Look in /sapphire/FileSystem/File.php. Instead of hacking this file, you can try defining your own list of acceptable extensions in /mysite/_config.php and add the djvu to them (I just copied the whole list, but you could customize it). That seems to do the trick...
File::$allowed_extensions = array(
Third: in the assets folder there is a web.config file, that also contains a list of extensions. As I understand it, this doesn't restrict uploads for other filetypes, but could prevent them being displayed in the frontend. Maybe you should add the extension here as well (haven't tested this)...