It's hard to tell exactly what is going wrong without being able to at least poke around inside the files you're having problems with, unfortunately. Just as not all ebook readers are created equal, not all ebook files are created equal. Images not showing up could be conversion errors as much as they could be problems with your device. It might not be a bad idea to check out the files on other reading systems (there are a myriad of free desktop reading systems, like the Chrome extension Readium, the Firefox EPUBReader add-on, and good old Adobe Digital Editions). If the images don't show up in those systems either, it's probably a problem in the ebook file rather than with your device.
For scientific texts, I would advise that you check to be sure the reading system supports MathML, which will allow properly formatted equations to show up as readable, searchable text. Note that for MathML to work, the manuscript must have been formatted properly, and since not all reading systems support MathML, not all publishers necessarily bother with it. Since this is a chicken/egg problem, I'd advocate picking up a MathML reader and making sure to harass publishers that don't use MathML until they do. You can use the support grid at epubtest.org to find MathML-capable systems, but most of that information is geared toward tablets.
For reflowing not working where there are formulas or graphics—do you mean that the images (and formulas that aren't MathML are frequently just images, to preserve formatting) aren't expanding in size when the font is increased, for example? That's something that you'll see in any ebook reader, I'm afraid. The best you can do there is get something with a good zoom feature, which unfortunately argues against eInk devices.
Display size is probably the easiest single criteria to look into, though it's also fairly subjective—what's fine for you to read my be too cramped for me. As mentioend above, zooming is going to be somewhat problematic on any eInk device, as the refresh rate for eInk is much slower than for a tablet screen. If tablets are absolutely not an option, you might want to look around for eInk readers with large displays; a quick search found me this article.
Best of luck with your search!
EDIT: Here's a good summary of which reading systems support MathML: http://www.mathjax.org/resources/epub-readers/