This article was prompted by this comment on my previous posting about disassembling the Toshiba Canvio 3Tb External Hard Drive:
Hi! Love your blog, haven’t had a reason to comment yet but here goes.
I recently bought some Hitachi 4TB USB3 external drives because they were (much) cheaper than buying the bare drives.
Unbeknownst to me the bridge board only supports advanced format disks so I’m wondering whether the VLI controllers on your drive’s bridge board might have the same limitation.
Thanks for your comment James. I replied with this long comment:
You do raise an interesting and rather “complicated” point which I might have to devote an entire blog post to once I confirm what’s happening. It’s rather technical – I apologize in advance if it’s a bit difficult to follow.
I will try to summarize, but it seems that the bridge chips that *are* capable of 2+Tb (i.e. they break the 2.1Tb barrier) are often “switched” to present the drive as 4k “native” sector size so that the drives can be (strangely) partitioned in MBR so that 3+Tb drives attached to Windows XP machines (which don’t understand GPT) via USB3.0 still operate properly. This works because the number of blocks in the drive doesn’t exceed the 32-bit size limit when you represent each block as 4k in size rather than 512 bytes.
The hard drives themselves are 4k “internally” but present 512e (i.e. 512 byte emulated sectors), so the Advanced Format drives are (at a low level) pretty much identical. They have specific information flags which can tell the OS they “prefer” transfers aligned and sized to 4k, but will still process 512 byte accesses just with some performance penalty. There are no drives which present native 4k sectors to the OS directly without 512e, at least, to my knowledge on the consumer market. Despite the drives themselves having information that signals to the OS what its “real” self is (i.e. 4k native, 512e), the bridge chips do not pass these commands – so instead, the OS relies on what the bridge chip reports (which will be 4k native).
The *magic* is really happening at the bridge chip level where some translation is happening, thus I believe that drives which are *not* AF and are used in computers directly (and partitioned with LBAs of 512 bytes) may have trouble when transferred to bridge chips which do the sector-lumping translation as the block numbers no longer align as each block is 8 times bigger than before and vice-versa. There may even be a slight funny-thing going on with sector slipping as normally, MBR does start the first partition at sector 63 (which isn’t going to be aligned with a straight mapping). This was one of the big issues with migrating to AF drives back in the “XP” days. I’m really only speculating with the sector slipping, but I’ve seen the sector size reporting as native 4k with my own eyes. The effects of this I will need to confirm by experiment. It does, however, bear strong relations to when we had capacity barriers which we had to break – especially in the days of 528Mb – 8.4Gb barriers where CHS translation was the way (i.e. swapping numbers between cylinders, heads, sectors to make a geometry which fits into the limitations).
The fun seems to happen in the bridge chip’s programming – many of these options can be set in the EEPROM but the utilities to do so are restricted to factory distribution, and may be password protected too. I know because I’ve seen the utility for the Asmedia chips which have the option, and no doubt other bridge chips might as well. Maybe if you could obtain the appropriate EEPROM utility, you can upgrade the firmware, and change the compatibility flags (amongst other things, like the VID, PID, and friendly name strings).
Another issue I have come across in certain bridge chips is timing compatibility issues, where replacing drives with others leads to unreliable detection of the drive, and thus it didn’t work at all. This happened with an Initio chipset when combined with a certain SSD, but I’m sure edge cases may pop up from time to time despite the well-meaning standards which do their best to maintain interoperability.
I would usually not recommend people to buy encased drives with any intention of “reusing” the bridge chip except for in an emergency – but if there are problems with the bridge chip, I agree, you really need to know before you need to use it and find it doesn’t work! After all, some also use the friendly name to “hide” the true device name of the drive which is placed within the case, just one of the “features” which you expect to see if you do these kinds of things.
I’ll do my best to explore and investigate – I’ve got several different bridge chipsets and some AF and some pre-AF drives, so I’ll see what my digging turns up .
As it turns out, I just couldn’t wait to get my hands dirty, because this is an important issue which doesn’t seem to have had much exposure at all. I just had to verify what I was expecting.
Bridge chips are ICs which convert between interfaces. In the early era of USB external drives, USB to IDE bridge chips were common. As an aside, they had limitations – all of them only recognized LBA capable drives (or optical drives) only. Attaching a 528Mb or lower CHS type drive to them would result in the bridge hanging and not reporting a capacity whatsoever. Further to this, some of the early bridges only supported 28-bit LBA as opposed to the 48-bit LBA presently used, thus not being compatible with drives >128GiB.
Nowadays, most external drives are being built with more modern USB to SATA bridge chips from various vendors. The important thing to note is that each different vendor’s products may behave differently – anything from performance differences to behavioural differences. Many chips also have configurable parameters which are configured at manufacture into an external EEPROM which affects how they identify themselves (say, branding) and also the enabled functionalities (possibly a >2Tb compatibility mode, UASP support etc). Changing these parameters requires knowing the bridge chip (disassembling the drive) and finding the manufacturer’s tools (which is sometimes very difficult). Other than that, the configuration may be blocked by a password as well, so it’s not a given that one can get the configuration that they wish.
Ever since SATA drives eclipsed the 2Tb mark, the number of 512 byte sectors have eclipsed the 32-bit limit. As a result, MBR partitioning is unable to address the whole capacity of the drive, and older operating systems and BIOSes had problems as they were only capable of booting from MBR and only capable of accessing MBR formatted drives. As a result, the 3Tb and greater drives are most compatible with newer OSes and motherboards that support UEFI booting.
It’s not the first time that we’ve hit capacity barriers and the solutions in the past have been kludgey and caused problems. There was translation algorithms which shuffled cylinders, heads and sectors around to make things fit, as well as overlay software which took over where the BIOS limitations were in place. Unfortunately, the experiences back then were that subtle bugs and differences in translation geometry meant that drives could not be accessed where the transplant computer’s BIOS had a different idea about the translation geometry. Furthermore, accessing the drive without the overlay software loaded could cause mangling of the drive’s data.
Have we just re-introduced this unfriendly situation?
Understanding Advanced Format
There’s a fair amount of literature about Advanced Format on the internet, so I won’t re-create everything. Instead, I will give you the most important points:
- Advanced Format was adopted by drive manufacturers to increase the size of a physical sector on the drive from 512 bytes to 4 kiB (i.e. 8 times larger).
- Advanced Format allows for better efficiency by reducing overheads from sector markers etc on the disk.
- All Advanced Format drives present themselves in 512e mode (i.e. they emulate 512 byte sectors for compatibility reasons). The sectors can be accessed in 512-byte chunks, although, if these accesses are not aligned to the 4kB sectors, there can be a performance penalty as the drive may have to rewrite two sectors. To the Operating System, they do not have to care although that may result in performance penalties.
- Native 4kB sector access has not hit the market yet, but is likely to be the future.
- Advanced Format drives can signal to the operating system that their physical sector size is 4kB so the OS can better manage accesses to the hard drive.
Identifying Advanced Format Drives
In order to identify Advanced Format drives, you need to check the label on the drive or look up the model number. Further to that, under Windows 7, you can use the fsutil fsinfo ntfsinfo c: (or whatever drive letter) command under an Administrative cmd prompt to find the Bytes per Sector (access) and Bytes per Physical Sector (reported by the drive).
For example, the drives in my main desktop box:
Here, we notice an incongruency – all of the 2Tb drives in this box are Advanced Format, however, all the Seagate drives report 512 bytes. Why? Well, it looks like that’s their choice as they have a “SmartAlign” technology which they claim, provides no extra steps to accommodating advanced format.
What does this all mean? It just means all the drives are 512e. They are all presenting themselves as 512 bytes per sector drives. Advanced format or not, it should NOT make any difference.
In practice, the 2Tb drives, AF or not, when connected to the VLI bridge, the drives all identify as 512 byte sectors and work just fine.
Of course, the physical sector size is not passed through the bridge as the commands are not directly supported.
What about 3Tb and greater?
All drives 3Tb and greater are advanced format drives. For example, here’s what the WD WD30EZRX 3Tb Green drive looks like when directly connected:
Note that aside from the larger number of sectors, the drive is 512e as per the earlier image.
This is where things get strange – attach this drive to the VLI bridge and things go wrong. The drive isn’t mounted – and the disk management console claims it’s unpartitioned:
When checking in WinHex, things become clearer –
The drive reports itself as 4kB sector size – as a result, each sector number LBA refers to each 4kB block rather than each 512 byte block. This means all of the partitioning information and allocation information does not make sense anymore! But can the drive be imaged and mounted “virtually” as a 512 byte block device? Possibly. The data looks like it’s there, and it doesn’t look like there’s any obvious sector mangling going on. I’ll leave this to someone else to find out as I’m too busy (especially dealing with the failure of a 2Tb boot drive).
But this isn’t a problem if you formatted the drive in the case and access it through the bridge. That would mount just fine …
This is another WD30EZRX that I own – notice how the same physical model of drive is being reported as 4096 bytes? This makes it appear as a 4k native drive and this is happening because of the bridge chip, not the drive!
As a result, if you format and use it through the bridge, it will work fine.
If you format and use the drive directly, it will work fine.
But if you move the drive formatted through the bridge to a computer, or from the computer to the bridge, it will not be mountable. The translation mangles the drive. BUT this only applies to drives larger than 2Tb. Drives which are 2Tb or smaller are mounted as 512 byte sectors on both a computer’s SATA controller and through the bridge.
What a mess! But the reasons are straightforward – there may be a limitation to the number of blocks a USB Mass Storage device can report, and older machines running Windows XP can now use these larger drives with MBR partitioning because the number of blocks (where each block is 4kB rather than 512 bytes) is now less than the 32-bit limit.
Lets try another controller …
I don’t have many USB to SATA bridge chips lying around – after all, I had very much abandoned USB external drives due to performance difficulties. But I had this bridge board from an old 2Tb Seagate Expansion Desktop drive –
This bridge board is based on the Initio INIC-1608L chipset. Plugging in any 2Tb or smaller hard disk, it works as expected, reporting 512 bytes allowing free interchange of the drive between internal SATA controllers and USB bridge boards.
Plugging in a 3Tb drive is trouble –
It gets stuck doing this for a while. Then, when it completes, nothing mounts, and it continually seeks (as if being continually being re-read due to read error).
WinHex confirms this – it just couldn’t access the drive at all. What does this mean? It means that older non-3Tb capable bridges which don’t do translation are not a solution to this problem as they just won’t work AT ALL!!!
Thanks very much James for alerting me to this issue that I had subconciously known of, but hadn’t determined and verified experimentally. This issue is definitely important for anyone with 3Tb or greater external drives – you won’t be able to remove the drive and plug it into a desktop computer and mount it directly. It would be best to bring it to another identical bridge chip just in case other chips have other ideas about how the translation should be done.
You may be able to image the whole thing and then mount the image on a loop device with a specified block size of 4096 bytes to make it work under Linux, although I have not tried and I wouldn’t know how to do it off the top of my head.
So I suppose those people having 3Tb+ drives on their desktop and are expecting to throw them into an external USB case for quick recovery or copying files back and forth – you probably can’t. Aside from the bridge not being able to handle >2Tb drives, the bridge may also do translations and hence screw everything up.
Just another way how the “venerable” old backwards compatibility of the IBM PC wreaks havoc with progress.
I hope this answers your questions James!