One reason could be the filesize: php can only upload up until a certain (php.ini) filesize (mostly about 2MB) It doen't know the filesize in advance, so upload will start, but then just not be completed. Follow a similare discussion here:
http://www.silverstripe.org/dataobjectmanager-module-forum/show/13243
Next: SiverStripe has predefined a hughe number of file extensions as acceptable for upload, but djvu is not part of them. Look in /sapphire/FileSystem/File.php. Instead of hacking this file, you can try defining your own list of acceptable extensions in /mysite/_config.php and add the djvu to them (I just copied the whole list, but you could customize it). That seems to do the trick...
File::$allowed_extensions = array(
'','html','htm','xhtml','js','css',
'bmp','png','gif','jpg','jpeg','ico','pcx','tif','tiff',
'au','mid','midi','mpa','mp3','ogg','m4a','ra','wma','wav','cda',
'avi','mpg','mpeg','asf','wmv','m4v','mov','mkv','mp4','swf','flv','ram','rm',
'doc','docx','txt','rtf','xls','xlsx','pages',
'ppt','pptx','pps','csv',
'cab','arj','tar','zip','zipx','sit','sitx','gz','tgz','bz2','ace','arc','pkg','dmg','hqx','jar',
'xml','pdf','djvu',
);
Third: in the assets folder there is a web.config file, that also contains a list of extensions. As I understand it, this doesn't restrict uploads for other filetypes, but could prevent them being displayed in the frontend. Maybe you should add the extension here as well (haven't tested this)...