Tomshardware botches Intel Atom energy efficiency tests badly

Update 8/20/2008 – I need to fix my mistake because I quoted some initial informal numbers from Jack that didn’t include the 3.5″ HDD or dual-threaded tests.  Now that the I have his full data set, I’m need to make some corrections.  I apologize for my mistake and I will fix it below, but my insistance that Tomshardware is off by a LOT has not changed though I was wrong about saying the Intel 945GC/Atom board could run on 802.3af.  Tomshardware on the other hand only did NOT correct the mistake despite acknowledging my emails to them, they went ahead and and published more results using an 850W power supply which is horrendously stupid.

For those who are going to say that Thorn who critized me in the comment section was right all along despite the fact that he did not run ANY tests, he was still wrong by a long shot because he insisted that the Tomshardware numbers aren’t that far off.  It turns out that Toms hardware was off by 25.6W from the Sparkle 220W power supply and my hasty post based on informal data that mistakenly excluded the hard drives was off by 11 W.  Let this be a lesson to me for posting informal data and once again, I apologize to my readers for my mistake.

After praising Tomshardware for doing a good job fixing their flash storage efficiency article, I must point out that Tomshardware’s made another horrendous error by claiming that and Intel D945GCLF system with an Intel Atom 1.6 GHz processor consumes 59W idle power.  Mr. Dandumont who authored the article claims that he used an “80 Plus” power supply implying that he was getting at least 80% efficiency on the PSU (Power Supply Unit) but that is a huge mistake.

UPDATE 7/21/2008 – Tomshardware’s Senior Editor Matthieu Lamelot has responded that they used the Tagan EasyCon U15 530 W power supply.  My 600W guess was fairly close and it means that Tomshardware loaded their PSU to roughly 3.6% load and that translates to roughly 30% efficiency which makes their benchmarks very inaccurate and misleading.  Tomshardware should at least test with a good 80 Plus 220W PSU but ideally they should test with a sub-100W PicoPSU.

The 80 Plus rating only claims greater than 80% efficiency if you’re talking about workloads between 20% to 100% output power and my guess is that Mr. Dandumont used a 600+ watt PSU which means he was likely loading the system at less than 3% workload.  3% workload on a PSU translates to a horrendous efficiency of less than 30%.

Tomshardware claims that the same system with a 7200 RPM 3.5″ hard drive consumes 59W idle and 62W peak using an energy efficient power supply but this isn’t even in the ballpark in terms of accuracy.  Tomshardware needs to correct this error and start using some more appropriate and smaller power supplies for testing computers that draw less than 50 watts of output power.

Update 8/20/2008 – My PhD friend Jack who is a very knowledgeable and meticulous tester tested the Intel D945GCLF with a Sparkle SPI220LE 220W PSU and got a measurement of 34W idle and 36.4W peak input power consumption.  Since the “80 Plus” SPI220LE gets around 73% efficiency at 19.2W output load according to Silent PC Review, we can reasonably estimate at 75% efficiency that the entire system actually consumes ~27W output power from the PSU.

Without the hard drive, the system peaked at 27.2W which means the output power from the PSU would be ~20W which is too much for 802.2af PoE.  The Intel 945GC chipset is unfortunately a bit power hungry so when the next version of Atom with on-die controller and graphics built in to the CPU, we can hopefully see the output power requirement dive down below 10W but until then, 802.3af is out of the question.

The SPI220LE is above 80% efficiency when the output load level is above 20% and here we’re only at around 8.1% load level so the efficiency drops.  I’ve done some testing with the Sparkle 55W Open Frame PSU and found that it runs at around 80% efficiency for the Intel 945GC/Atom board despite the fact that it’s not 80 Plus certified.  This is because the 55W PSU is running at optimum load levels.

Tomshardware redeems themselves with new flash versus hard drive energy efficiency tests

The last time Tomshardware posted an article on SSD Flash drives versus HDD Hard Drive power consumption, I gave them a well deserved “F”.  They had botched the analysis badly and drew incorrect hypothesis because they didn’t test their theories and it turned out that their theories were wrong.  But to be fair to Tomshardware, they took the criticism and came back with a much better piece of work and tested some of the suggestions I made such as fixed storage workloads across the different test beds.  This time I’ll give them an “A-” for effort and thoroughness so I recommend that you check out the interesting results.

However, it’s clear that their new conclusion conflicts with their old conclusion.  Their old conclusion claimed that flash drives resulted in lower battery life.  As it turns out, the hard drive they tested last time was one of the most efficient hard drives on the market and it wasn’t representative of the average hard drive.  This time they tested a newer SSD flash drive which resulted in remarkable performance and energy efficiency that completely dominated hard drives on all tests.  It’s kind of like saying that we scored the game 55 to 45 instead of 45 to 55.  Just imagine if they had scored a boxing match or basketball game this way and declared the wrong winner and then said “oh, but we were close on the actual score!”  So while I applaud Tomshardware for owning up to the mistake, I have to take some issue with the latest conclusion.

The correct conclusion – which I stated in my review of the first article – is that the average SSD flash drive does consume less energy than average Hard Drive though there are exceptions to this rule.  Furthermore, SSDs almost always achieve higher work per unit energy.

So does this mean that SSD flash drives are a better technology?  With the latest SSD drives like the OCZ SATA II 2.5″ product, the answer is a resounding yes.  Does that make it worth the money?  If high performance, complete silence, and the highest energy efficiency matters to you, then by all means it is worth the price premium.  If what you’re looking for is reasonable power consumption, low cost, and high capacity, then mechanical hard drives remain the best solution today.  The price of SSD flash drives were outrageous but the price has recently dropped dramatically and now you can soon pick up a 64 GB OCZ SATA II 2.5″ SSD for $280.

While that’s still about 5 times more expensive, it’s a very affordable price for a mid- to high-end laptop.  Prices will continue to drop and it’s possible that flash drives will eventually become mainstream in the laptop market if the price drops continue at the rate they’re falling today.  We will also see smaller 1.8″ SSD models or simply a “chip” that gets plugged in to an ultra portable laptop so the size and durability is where flash technology shines.

Now if I can just get Tomshardware to fix their badly botched Intel Atom power consumption analysis.

Testing out Microsoft Expression Design 2

For a long time now, I have been looking for a quality, affordable photo/image editor. This weekend I gave Microsoft Expression Design 2 a try. The upshot? It is missing some really critical features, the UI focuses a bit too much on form and not function, and the price point is poor.

Granted, I am comparing it against Photoshop (Photoshop CS 1, at that, which is why I am in the market for a cheap Photoshop replacement). But Microsoft bought the basis of this product (“Acrylic”) ages ago; it has been in development for a long time. One of the problems that it suffers from, is that is never figured out if it wanted to be a vector tool like Illustrator, or a raster tool like Photoshop. So it ends up with some really intresting ways of doing old tasks. For example, the “paintbrush” actually constructs a vector path, and “closing” the brush generates a full shape object. I am sure that vector tools work this way (I don’t have any experience with them), but considering that I was looking for a Photoshop stand-in, this was novel. But at the same time, I can’t find a magic wand, an eraser, or a pencil tool. There is no “gradient” tool; a “gradient” is merely a type of fill that can be applied to shapes. There is no mask tool either, but my suspicion is that you are supposed to use shapes for masking as well as outright design pieces.

I really like how the “special effects” work, though. You apply them on to individual objects or layers. And you can change the order that they are applied. The end result is that what Photoshop calls a “filter” never touches the underlying pixels directly, which gives you some great flexibility. But the lack of raster editing features can’t make up for this in my mind.

The UI deserves special mention. It is gorgeous. It is pretty intuitive. But it has some odd quirks that kill me, like having a toolbox be too tall so it has scroll bars, and to make it worse, the drop down boxes in it go past the part on screen, so you need to scroll the toolbox in order to see the bottom of the drop down within it. Huh? Why not have the drop down limit itself to the visible portion of the toolbox? Another oddity is that many widgets rely upon “gestures”. To rotate a gradient, you can either type in a rotation angle, or you can “draw a circle” with the mouse while holding the left mouse button down over the angle type-in box, to adjust the angle. Again, huh? Many other numbers can be adjusted with “gestures” too, over UI elements that are actually sliders, but do not appear as such. On the other hand, the UI was very smart; “skew” is at the same level as resize or rotate, in terms of, “bring the mouse near a point on a shape and you can do it.” But the places where it was smart hardly compensated for its drawbacks. Most of the UI drawbacks come from trying to make a pretty design that tramples the standard widgets toolbox.

Finally, the price. Expression Design 2 is not available standalone, although its demo it. If you want to buy it, you need to purchase the full Expression Studio 2 product. Granted, it is part of the suite, and it is geared towards the XAML/WPF/Silverlight stuff that Microsoft is pushing. And yes, that suite costs about what a full copy of Photoshop CS 3 costs (which is a bit more than the full Corel suite). But unless your plan is to dive deep into XAML/WPF/Silverlight, this product is not worth buying the suite for just to have this one product (the suite itself is a good price, in my opinion, especially since Expression Web 2 is a great product). Compared to Photoshop, this is not a very good raster editor at all. Indeed, I think that the free Paint.Net is a much better choice for standard image/photo editing. I can’t compre it to any vector tools, but I suspect that it won’t matchup well against them either.

J.Ja

Horrible analysis claims flash drives consume more power

 

Update 7/14/2008 - Tomshardware redeems themselves with new flash versus hard drive energy efficiency tests

Lots of people are talking about Tomshardware’s latest claim that SSD (Solid State Disk) flash drives are less energy efficient than a conventional laptop hard drive.  The problem is that the analysis is fundamentally shoddy.

Tomshardware’s Patrick Schmid and Achim Roos claims that a computer with a SanDisk SSD5000 SSD drive doesn’t last as long as the same computer running a conventional Hitachi 7K200 2.5″ 7200 RPM hard drive.  We can pretty much throw this conclusion out the window because the SanDisk SSD5000 has a PEAK power consumption of 1 watt while the Hitachi 7K200 has an IDLE power consumption of 1.1 watt.  So even if we assume that the test they ran was so flawed that it forced the SanDisk to operate at PEAK and permitted the Hitachi to operate at IDLE, the computer running the SanDisk SSD should still consume less power.  But the Hitachi 7K200 actually has a PEAK power consumption of 3.2 watts so the odds are it consumed a lot more than 1.1 watt during the tests and the battery drain test run by Schmid and Roos are flawed.

It’s actually quite simple to explain how they might have botched the experiment.  The SSD drives are capable of far more performance especially when it comes to random access.  By having more storage IO performance, it’s very easy for the CPU and the rest of the system to work harder because they’re not idling as much waiting for data from the storage system.  Whenever you’re doing a batter drain test, you have to keep the variables in check.  The amount of work done by the two systems must be identical because cranking one system’s CPU to a higher utilization rate will easily cause it to consume more power and run the battery down.  The ideal test for a battery drain test is to play back a movie using popular movie formats and movie software.  This way, the computational workload on the CPU, graphics, and the storage system is identical across all the systems and you’re not unfairly stressing one system more than the other.

Schmid and Roos are so confident that they state “No, our results are definitely correct” and they have theories to explain why they’re think they’re right.  On hard drive power consumption, the Tomshardware duo explains that for sequential access, hard drives should “not require much more than the idle power” because the mechanical actuators aren’t jumping back and forth.  That’s an interesting hypothesis, too bad they never tested that theory before they exclaimed to the world that their conclusions are definitive.

Xbit Laboratories did test power consumption between idle, random access, and sequential access but they showed that this theory is wrong.  In fact, power consumption between random access and sequential access is almost identical and in some cases, sequential access actually consumed more power.  How can this be?  Well with random access, the hard drive is putting out maybe 1 Megabyte/second of data which isn’t very much work for the I/O (Input/Output) logic on the storage device.  With sequential access, the hard drive is putting out roughly 50 MB/sec which stresses the hell out of the IO logic on the hard drive.  This additional power consumption in the IO logic is enough to offset or even exceed the power consumed by the mechanical actuators.

However, Schmid and Roos proclaimed in bold subtitles that “Flash SSDs only Know Two Power States” and that SSDs are “active or idle”.  Oh really?  Is it possible they concluded this because they only tested two working states rather than testing three working states between idle, random access, and sequential access?  I’ve been looking and analyzing power consumption characteristics for a long time and I’ve never seen IO logic with only two power states.  IO logic like all microprocessors subtly increases in power consumption as you increase workloads.  Tomshardware would have you believe that an SSD would consume 0.5 watts in idle and immediately jump to 2.9 watts as soon as you throw any work at it.  So they’re claiming that an SSD pushing out data at 1 MB/sec would consume just as much power as it would at 100 MB/sec.  This is a preposterous hypothesis and it is laughable.

Tomshardware only tested “load” (whatever that means since they didn’t disclose methodology) and idle power consumption.  The problem is that they don’t factor in the fact that the SSDs are being forced to operate at above 100 MB/sec while the hard drive is only operating at around 50 MB/sec.  In fact if you divide their Megabyte/second measurements by the wattage (Joules/second) measurements, you can actually get the megabytes per joule (unit energy) number.  This is the number of megabytes transferred using a single joule of energy.  I made an assumption and used their average sequential throughput divided by load power measurements and produced the following table showing how much more efficient the flash drives are if you account for the amount of work being done.

The proper way to do this test is to run all the devices at equal throughput.  So you could simply play a DVD movie off the storage device at 1 MB/sec (8 megabits/sec) or an HDV stream at 3 MB/sec and measure the power consumption at these fixed storage workloads rather than assuming that the SSDs “only know two power states”.

The purpose of my analysis is not to say you should buy SSD drives for your laptop.  I simply abhor bad science and bad analysis especially when it gets taken seriously.  I’m actually not a big proponent of SSD drives in laptops yet because the economics aren’t there *YET* and the power consumption difference isn’t that great.

At this point in time, SSDs make sense for submarine duty in the US Navy where silence means survival.  It might also make sense in ultraportable laptops where you don’t need that much capacity and the SSD saves space and some power.  Companies like MSI are actually opting out of SSDs for their 10 inch “Wind” notebook because of the additional storage capacity and the power savings aren’t all that compelling.  SSDs also make sense for server duty where random access performance is extremely important.

The reality is that the average SSD drive doesn’t save all that much power compared to a conventional 2.5″ hard drive and it’s quite possible that some 2.5″ hard drives are more energy efficient than some SSDs.  But the analysis and conclusions drawn by Tomshardware are not to be taken seriously.

Blu-ray PC blues

In an effort to build the cheapest computer I can build that plays Blu-ray DVD movies, I put the NVIDIA 7150 chipset with Intel Allendale dual-core 2.93 GHz (overclocked) test bed to work.  I’ve already been having some problems with this inexpensive embedded NVIDIA chipset but I wanted to put it to the most challenging video load it can handle.

To start off, I put in a Pioneer BDC-S02BK Blu-ray ROM drive in the computer and put it to the test using PowerDVD 8.0 trial edition.  The result was an absolute failure and I couldn’t even get the drive to recognize “The Other Boleyn Girl” 2008 Blu-ray edition that I rented from Netflix.  The drive would recognize HQV Blu-ray edition test disk but the frame rate was just awful and at times it was like watching a 5 fps slide show and the CPU hit near 100% for both CPU cores on the Intel Allendale 2.93 GHz processor.  Mid way through, PowerDVD 8 crashed and took about 5 minutes to terminate the process.

So I went to Fry’s and picked up a Lite-on BD-ROM drive for $110 and that came with an OEM edition of PowerDVD 7 which only permits 2-channel audio but that was fine for me since I’m only hooking the computer up to my DLP HDTV.  I put the disk in and “The Other Boleyn Girl” worked with PowerDVD 7 but it was still jerky at times and the CPU hit 90+ percent again.

At this point since this motherboard and embedded GPU is absolutely worthless for Blu-ray, I’m going to try and pick up an AMD Radeon HD 3450 PCI-Express graphics card with full H.264 and VC-1 acceleration for $37.  I’m going to switch to an inexpensive Intel G33 chipset motherboard because I want the S3 sleep state to work so that the computer can have instant on capability.

The NVIDIA 7150 motherboard is crashing whenever it wakes from S3 sleep state and I don’t know if it’s a bad implementation by XFX or if all NVDIA 7150 chipset motherboards have this problem.  Since I can’t get any support on this from NVIDIA or XFX after repeated queries, I’m going to assume the worst until I get my hands on another 7150 to test and suspend my recommendation of this chipset for now even if you’re not using it for Blu-ray playback.

Why are video drivers so hard to get right?

I (finally) wrapped up a 6 or 7 month long struggle with my video card. See, some time ago, some update or another did something… odd… with my video card (an NVidia 7600 GS, for the record, the best passively cooled, dual DVI output card I could find when I put this PC together). It started doing this “thing” where major portions of the screen would be transparent to the layer beneath. Oddly enough, it only occured when a game was running full screen. Let me tell you, it was quite odd going through Half Life 2 (I know, I am WAY behind the times) being able to see through walls if I get right next to them. In many situations, there would be some odd “snow” in areas too. To make it even more strange, the games would work fine the first time they were run after a reboot, or if I started the game in a window and then told it to run full screen. Needless to say, it really sucked.

For the last umpteen months, I would occassionally fool with it for a few hours… messing with settings, updating drivers, searching the Internet. A friend of mind reported that he had an identical problem. I was loathe to revert to a much older driver, because I had severe performance problems with some of them. I kept meaning to purchase another video card, but frankly, I find purchasing desktop-grade hardware that is paid for from my bank account to be quite stressful. I’ll price out $10k worth of server kit in a few hours, including vendors and SKUs, but for some reason, personal PC equipment turns me into a neurotic diva. Of course, for the money I could have earned in that time I spent wrangling with the existing card and picking out a new card, I could have bought a whole new PC…

I finally picked out a suitable replacement, a low end Sapphire Radeon card (no way was I going back to NVidia). Ironically, I purchased the NVidia card because ATI cards have had so many driver problems over the last few years. For my needs, which are quite low end (a few older games, driving a pair of monitors for typical business usage), I don’t need anything fancy. I need it to not cause me headaches.

I finally solved my problem this weekend, by dicovering that NVidia driver installers leave themselves on my drive, and I found a driver from before the problem. After installing it, it worked like a charm.

But I find the whole situation rediculous. ATI and NVidia have both had very long (as in, “a few years”) periods where they had a bad driver reputation. Imagine if, say, drive controllers were as unreliable as video drivers. You know, doing things like resetting the driver and “going blank” for five minutes at a time (my previous ATI card did this, to solve the problem I’d use the keyboard to get the PC to sleep, then wake it back up) with no warning. I am sure that people would love their hard drives doing that. Or if waving the mouse around too fast caused the system to blue screen, blaming a failure in the mouse driver. Yeah, that’s going to move a lot of units. I can see the computing public loving to have to search out drivers for their sound card because their current card does not play Outlook’s new mail notification sound properly, due to a driver incompatbility with Outlook (but not Word or Excel).

Now, I know that there is a certain reality here. Video drivers are very complex. They involve a lot of fancy algorithms, optimizations, and so on. They need to be compatable with a ton of applications and APIs which are doing some tricky things. The hardware operates under some very tight constraints. But you know what? So does an F-15. If F-15′s had drivers written by ATI or NVidia, the US Air Force could be pwn3d by the local RC airplane club. Seriously.

NVidia, ATI… get with the program. This situation is only possible because you two have a duopoly. Someone’s going to eat your lunch someday, and bad drivers are the worst form of customer service possible. I’d jump for the chance to give a 3rd party my video card dollar (as little as I spend on them) provided that the drivers weren’t junk.

J.Ja