Experiment: USB to SATA bridge chips and >2Tb drives

This article was prompted by this comment on my previous posting about disassembling the Toshiba Canvio 3Tb External Hard Drive:

Hi! Love your blog, haven’t had a reason to comment yet but here goes.

I recently bought some Hitachi 4TB USB3 external drives because they were (much) cheaper than buying the bare drives.

Unbeknownst to me the bridge board only supports advanced format disks so I’m wondering whether the VLI controllers on your drive’s bridge board might have the same limitation.

Regards,
James

Thanks for your comment James. I replied with this long comment:

Dear James,

You do raise an interesting and rather “complicated” point which I might have to devote an entire blog post to once I confirm what’s happening. It’s rather technical – I apologize in advance if it’s a bit difficult to follow.

I will try to summarize, but it seems that the bridge chips that *are* capable of 2+Tb (i.e. they break the 2.1Tb barrier) are often “switched” to present the drive as 4k “native” sector size so that the drives can be (strangely) partitioned in MBR so that 3+Tb drives attached to Windows XP machines (which don’t understand GPT) via USB3.0 still operate properly. This works because the number of blocks in the drive doesn’t exceed the 32-bit size limit when you represent each block as 4k in size rather than 512 bytes.

The hard drives themselves are 4k “internally” but present 512e (i.e. 512 byte emulated sectors), so the Advanced Format drives are (at a low level) pretty much identical. They have specific information flags which can tell the OS they “prefer” transfers aligned and sized to 4k, but will still process 512 byte accesses just with some performance penalty. There are no drives which present native 4k sectors to the OS directly without 512e, at least, to my knowledge on the consumer market. Despite the drives themselves having information that signals to the OS what its “real” self is (i.e. 4k native, 512e), the bridge chips do not pass these commands – so instead, the OS relies on what the bridge chip reports (which will be 4k native).

The *magic* is really happening at the bridge chip level where some translation is happening, thus I believe that drives which are *not* AF and are used in computers directly (and partitioned with LBAs of 512 bytes) may have trouble when transferred to bridge chips which do the sector-lumping translation as the block numbers no longer align as each block is 8 times bigger than before and vice-versa. There may even be a slight funny-thing going on with sector slipping as normally, MBR does start the first partition at sector 63 (which isn’t going to be aligned with a straight mapping). This was one of the big issues with migrating to AF drives back in the “XP” days. I’m really only speculating with the sector slipping, but I’ve seen the sector size reporting as native 4k with my own eyes. The effects of this I will need to confirm by experiment. It does, however, bear strong relations to when we had capacity barriers which we had to break – especially in the days of 528Mb – 8.4Gb barriers where CHS translation was the way (i.e. swapping numbers between cylinders, heads, sectors to make a geometry which fits into the limitations).

The fun seems to happen in the bridge chip’s programming – many of these options can be set in the EEPROM but the utilities to do so are restricted to factory distribution, and may be password protected too. I know because I’ve seen the utility for the Asmedia chips which have the option, and no doubt other bridge chips might as well. Maybe if you could obtain the appropriate EEPROM utility, you can upgrade the firmware, and change the compatibility flags (amongst other things, like the VID, PID, and friendly name strings).

Another issue I have come across in certain bridge chips is timing compatibility issues, where replacing drives with others leads to unreliable detection of the drive, and thus it didn’t work at all. This happened with an Initio chipset when combined with a certain SSD, but I’m sure edge cases may pop up from time to time despite the well-meaning standards which do their best to maintain interoperability.

I would usually not recommend people to buy encased drives with any intention of “reusing” the bridge chip except for in an emergency – but if there are problems with the bridge chip, I agree, you really need to know before you need to use it and find it doesn’t work! After all, some also use the friendly name to “hide” the true device name of the drive which is placed within the case, just one of the “features” which you expect to see if you do these kinds of things.

I’ll do my best to explore and investigate – I’ve got several different bridge chipsets and some AF and some pre-AF drives, so I’ll see what my digging turns up :) .

– Gough

As it turns out, I just couldn’t wait to get my hands dirty, because this is an important issue which doesn’t seem to have had much exposure at all. I just had to verify what I was expecting.

Semi-related History

Bridge chips are ICs which convert between interfaces. In the early era of USB external drives, USB to IDE bridge chips were common. As an aside, they had limitations – all of them only recognized LBA capable drives (or optical drives) only. Attaching a 528Mb or lower CHS type drive to them would result in the bridge hanging and not reporting a capacity whatsoever. Further to this, some of the early bridges only supported 28-bit LBA as opposed to the 48-bit LBA presently used, thus not being compatible with drives >128GiB.

Nowadays, most external drives are being built with more modern USB to SATA bridge chips from various vendors. The important thing to note is that each different vendor’s products may behave differently – anything from performance differences to behavioural differences. Many chips also have configurable parameters which are configured at manufacture into an external EEPROM which affects how they identify themselves (say, branding) and also the enabled functionalities (possibly a >2Tb compatibility mode, UASP support etc). Changing these parameters requires knowing the bridge chip (disassembling the drive) and finding the manufacturer’s tools (which is sometimes very difficult). Other than that, the configuration may be blocked by a password as well, so it’s not a given that one can get the configuration that they wish.

Ever since SATA drives eclipsed the 2Tb mark, the number of 512 byte sectors have eclipsed the 32-bit limit. As a result, MBR partitioning is unable to address the whole capacity of the drive, and older operating systems and BIOSes had problems as they were only capable of booting from MBR and only capable of accessing MBR formatted drives. As a result, the 3Tb and greater drives are most compatible with newer OSes and motherboards that support UEFI booting.

It’s not the first time that we’ve hit capacity barriers and the solutions in the past have been kludgey and caused problems. There was translation algorithms which shuffled cylinders, heads and sectors around to make things fit, as well as overlay software which took over where the BIOS limitations were in place. Unfortunately, the experiences back then were that subtle bugs and differences in translation geometry meant that drives could not be accessed where the transplant computer’s BIOS had a different idea about the translation geometry. Furthermore, accessing the drive without the overlay software loaded could cause mangling of the drive’s data.

Have we just re-introduced this unfriendly situation?

Understanding Advanced Format

There’s a fair amount of literature about Advanced Format on the internet, so I won’t re-create everything. Instead, I will give you the most important points:

  • Advanced Format was adopted by drive manufacturers to increase the size of a physical sector on the drive from 512 bytes to 4 kiB (i.e. 8 times larger).
  • Advanced Format allows for better efficiency by reducing overheads from sector markers etc on the disk.
  • All Advanced Format drives present themselves in 512e mode (i.e. they emulate 512 byte sectors for compatibility reasons). The sectors can be accessed in 512-byte chunks, although, if these accesses are not aligned to the 4kB sectors, there can be a performance penalty as the drive may have to rewrite two sectors. To the Operating System, they do not have to care although that may result in performance penalties.
  • Native 4kB sector access has not hit the market yet, but is likely to be the future.
  • Advanced Format drives can signal to the operating system that their physical sector size is 4kB so the OS can better manage accesses to the hard drive.

Identifying Advanced Format Drives

In order to identify Advanced Format drives, you need to check the label on the drive or look up the model number. Further to that, under Windows 7, you can use the fsutil fsinfo ntfsinfo c: (or whatever drive letter) command under an Administrative cmd prompt to find the Bytes per Sector (access) and Bytes per Physical Sector (reported by the drive).

For example, the drives in my main desktop box:

ntfsinfo1

Here, we notice an incongruency – all of the 2Tb drives in this box are Advanced Format, however, all the Seagate drives report 512 bytes. Why? Well, it looks like that’s their choice as they have a “SmartAlign” technology which they claim, provides no extra steps to accommodating advanced format.

What does this all mean? It just means all the drives are 512e. They are all presenting themselves as 512 bytes per sector drives. Advanced format or not, it should NOT make any difference.

In practice, the 2Tb drives, AF or not, when connected to the VLI bridge, the drives all identify as 512 byte sectors and work just fine.

ST2000DL003-9VT166-EXTERNAL

Of course, the physical sector size is not passed through the bridge as the commands are not directly supported.

What about 3Tb and greater?

All drives 3Tb and greater are advanced format drives. For example, here’s what the WD WD30EZRX 3Tb Green drive looks like when directly connected:

EZRX-native

Note that aside from the larger number of sectors, the drive is 512e as per the earlier image.

This is where things get strange – attach this drive to the VLI bridge and things go wrong. The drive isn’t mounted – and the disk management console claims it’s unpartitioned:

3Tb-via-bridge

When checking in WinHex, things become clearer –

3Tb-invoke-translation

The drive reports itself as 4kB sector size – as a result, each sector number LBA refers to each 4kB block rather than each 512 byte block. This means all of the partitioning information and allocation information does not make sense anymore! But can the drive be imaged and mounted “virtually” as a 512 byte block device? Possibly. The data looks like it’s there, and it doesn’t look like there’s any obvious sector mangling going on. I’ll leave this to someone else to find out as I’m too busy (especially dealing with the failure of a 2Tb boot drive).

But this isn’t a problem if you formatted the drive in the case and access it through the bridge. That would mount just fine …

3Tb-in-case-working-as4096native

This is another WD30EZRX that I own – notice how the same physical model of drive is being reported as 4096 bytes? This makes it appear as a 4k native drive and this is happening because of the bridge chip, not the drive!

  • As a result, if you format and use it through the bridge, it will work fine.

  • If you format and use the drive directly, it will work fine.

  • But if you move the drive formatted through the bridge to a computer, or from the computer to the bridge, it will not be mountable. The translation mangles the drive. BUT this only applies to drives larger than 2Tb. Drives which are 2Tb or smaller are mounted as 512 byte sectors on both a computer’s SATA controller and through the bridge.

What a mess! But the reasons are straightforward – there may be a limitation to the number of blocks a USB Mass Storage device can report, and older machines running Windows XP can now use these larger drives with MBR partitioning because the number of blocks (where each block is 4kB rather than 512 bytes) is now less than the 32-bit limit.

Lets try another controller …

I don’t have many USB to SATA bridge chips lying around – after all, I had very much abandoned USB external drives due to performance difficulties. But I had this bridge board from an old 2Tb Seagate Expansion Desktop drive –

Seagate Expansion Bridge Board

Seagate Expansion Desktop Bridge Board Bottom

This bridge board is based on the Initio INIC-1608L chipset. Plugging in any 2Tb or smaller hard disk, it works as expected, reporting 512 bytes allowing free interchange of the drive between internal SATA controllers and USB bridge boards.

Plugging in a 3Tb drive is trouble –

Initio-3Tb-Stuck

It gets stuck doing this for a while. Then, when it completes, nothing mounts, and it continually seeks (as if being continually being re-read due to read error).

Initio-3Tb-Error

WinHex confirms this – it just couldn’t access the drive at all. What does this mean? It means that older non-3Tb capable bridges which don’t do translation are not a solution to this problem as they just won’t work AT ALL!!!

Conclusion

Thanks very much James for alerting me to this issue that I had subconciously known of, but hadn’t determined and verified experimentally. This issue is definitely important for anyone with 3Tb or greater external drives – you won’t be able to remove the drive and plug it into a desktop computer and mount it directly. It would be best to bring it to another identical bridge chip just in case other chips have other ideas about how the translation should be done.

You may be able to image the whole thing and then mount the image on a loop device with a specified block size of 4096 bytes to make it work under Linux, although I have not tried and I wouldn’t know how to do it off the top of my head.

So I suppose those people having 3Tb+ drives on their desktop and are expecting to throw them into an external USB case for quick recovery or copying files back and forth – you probably can’t. Aside from the bridge not being able to handle >2Tb drives, the bridge may also do translations and hence screw everything up.

Just another way how the “venerable” old backwards compatibility of the IBM PC wreaks havoc with progress.

I hope this answers your questions James!

About lui_gough

I’m a bit of a nut for electronics, computing, photography, radio, satellite and other technical hobbies. Click for more about me!

This entry was posted in Computing and tagged , , , . Bookmark the permalink.

21 Responses to Experiment: USB to SATA bridge chips and >2Tb drives

  1. Czerno says:

    I’ve got an USB “enclosure” – the disk inside is a 1-TB only Hitashi 512 or 512e,
    but the included sATA – USB bridge (‘oxbridge’) still presents 4;096 nyte sectors
    to the host ! I whish I could re-program the bridge so it passes 512-sectors thru,
    but as you noticed, manufacturers – neither the bridge’s nor the enclosure’s -will
    provide a utility nor documentation to an end user. Grrr…

    Excellent article ! If you hear of someone having ‘reversed’ the chips and/or a way
    to acquire factory reprogramming utilities, please do share !

  2. Adolfo says:

    Great article. It is the most close thing I have found in the internet to try to fix my problem. I have a HD with 3TB in one only partition I was using with a VLI VL701 USB 3.0 to SATA II Bridge Controller temporarily until I recieved a HD enclosure. I wrote important data in there, but the chip failed before the enclosure arrived. Then, I put the HD inside the computer to try to access the data and it did not work, showing several partitions (3 in windows, 2 in linux), and when the enclosure arrived I tried again but the result was the same.

    What I am not sure after reading your article is if I can recover the data or not by connecting the HD into another VLI VL701 USB 3.0 to SATA II Bridge Controller.

  3. Pingback: Partition lost !! Please help me!!

  4. AlienTech says:

    Well I have many segate externals and it seems even those from 2010 on work fine. IE I can use the 3TB drive in a computer even to boot.. And access files on them outside the computer in the external case. The USB3 bridge ofcourse works fine but I did not think the old USB2 would work but it seems it does. So they have been working on this for a long time it seems. I have not tested it with WD or some others. Just have to make sure the partitions are not greater then 2TB. I made mine into a 2 TB and a 1TB to not have problems and made it GPT.

  5. Pingback: Review, Teardown: Astone ISO DOC-130 2.5″ & 3.5″ SATA II to USB 3.0 Dock | Gough's Tech Zone

  6. imz says:

    my external seagate 3tb expansion usb interface was damaged. tried to plugged directly but as expected read as raw 349.31gb..format the hd is not an option as i have a lot of important data on the disk (doc/pict/vid etc)..

    do you think that i can use the external base from this model in order to detect the hard disk correctly?a similar pcb board (raptor v2 VOR2 3.5″) on ebay is very expensive..i rather have a new extra hardisk+matching usb adapter.

    http://www.ebay.com/itm/Seagate-Backup-Plus-Desktop-3-TB-External-STCA3000101-Hard-Drive-695549-/271911733610

    • lui_gough says:

      Unfortunately, I can’t help you there. There is no easy way to tell which bridge chips perform the strange 8:1 sector translation, and which ones don’t just by looking. Your best bet is probably to match the model exactly to be sure the translation is performed identically (so there is no loss of data or corruption).

      By the sounds of it, your drive was written through an interface that performed the translation, so you will need to find one that does to access the data properly. If someone has that unit, they could probably check it using the fsutil command in Windows to determine whether it identifies as 512 bytes or 4096 bytes per sector to see whether it does translate the sector size, but that still won’t necessarily guarantee the translation is performed in an identical manner.

      – Gough

    • lui_gough says:

      Another possibility is that you might have plugged it into a port on a computer running a 32-bit Windows XP, or operating system and driver combination that does not support >2.1Tb hard drives, because you report it as 349.31Gb (i.e. a lot less than is expected). As a result, they cannot read the drive properly. Maybe it is a good idea to ensure you try it on a 64-bit machine running a 64-bit OS with up to date SATA controller drivers that is known to work properly with >2Tb hard drives.

      – Gough

      • imz says:

        Thanks for the detailed explanation!! Actually my hard drive detected as disc 0 = total 2794.52gb basic. There are 3 other partitions = 349.31gb RAW primary, 1698.69gb unallocated and lastly another 746.52gb unallocated. I’m using win 7 64bit, ASUS P8P67 LE flashed to latest bios. I will try to update SATA controller driver and update to you later. Thanks again!!

        • lui_gough says:

          As the partitions appear to be mangled, my bet is that your USB bridge board was configured for 8:1 translation, and you might have no choice but to use another board that performs the same translation to ensure the recovery of your data.

          That being said, the data does have the same structure, but since references in the data are by number of “blocks” – if the block size changes, all of the references are invalid. However, this might mean you can buy another external drive with 8:1 translation and just use Linux with dd to make a raw copy of the data from the extracted drive plugged into an internal port to the new USB drive – and if it is just an 8:1 translation with no other fancy manipulation, the copy might work without you having to destroy the new drive (i.e. take it apart). But if you’re not careful with dd, you could end up destroying your data as well (so be careful with your if= and of= declarations).

          – Gough

          • im says:

            hi,just to share that I’ve solved the problem by purchasing this HD docking system instead..ezdisk EZ0330…512k-4k sector translation switch on the fly 🙂

  7. quasar says:

    Hi Gough,
    thanks for this post,
    I’m buying a new WD 6TB, do you know any usb/sata bridge that works
    for this drive?
    Do I need to format this new drive? I need windows 7 to write into, but
    I’d like a performance filesystem.

    • lui_gough says:

      Most modern bridge chips will work with >2Tb drives, but their limits are unknown as of yet. I haven’t tried qualifying bridge chips with 6Tb or 8Tb drives, but suffice it to say, some must exist as Seagate do sell 8Tb external units, however, they do have the 4k sector translation going on, which can limit compatibility of some software expecting 512k sectors and prevent a “direct” drive plug-in to the system for data recovery. Some will have the awkward 4k-translation setup, whereas older ones often not claiming more than 2Tb support will end up presenting the 512-byte sector device correctly, but their ability to not corrupt the data above a certain threshold needs to be qualified and checked for your application as there may be some sort of roll-over due to hardware limitations (common with prior capacity limits – e.g. 28-bit LBA).

      Generally, bare drives supplied for installation into a PC will come unformatted. Partitioning and formatting is performed by the end user. For Windows, you really don’t have much choice especially if you wish to do it within Windows. Generally, for best performance and most features, you should stick with NTFS. The only other workable alternative by default on Windows would be exFAT which was more designed for use with large memory cards and is not as feature rich as NTFS, but does reduce host system requirements when it comes to working with the filesystem.

      – Gough

  8. Oliver says:

    Great writeup, thanks for shedding light onto the mechanics behind this.

    But one question remains: How do I then reformat the previously external drive and use it internally? My 5TB WD Elements turns out as a WD Blue drive, in Windows 10 as well as OS X it appears as two seperate drives (yes, disks) with one partition each.

    I have tried killing the partition table on both, using diskutil, fdisk, dd and some other tools. I just can’t persuade the drive to appear as one drive.

    Do you have any ideas? (I have four sealed WD elements lying around and was hoping to upgrade my NAS 😉

    • lui_gough says:

      I have not come across such a drive, so I can’t really offer any advice. It may well be that it has some very strange firmware for security encryption reasons to work with the original bridge chip and included WD software. I’m afraid you will have to consult someone else who has used one of these units.

      – Gough

  9. Robbie says:

    Finally someone prepared to explain the smoke and mirrors of the makers

  10. rob daw says:

    Please direct me to the right place if this is left field
    How do commercial external drives above the 2.2tb limit
    appear as MBR in windows 7/8/10
    I assume it has something to do with sneaky translation chips
    buried in the usb logic board inside the case
    I too have done experiments
    put a WD2t drive in a WD 1.5t case ….all ok
    put a WD3t drive in a WD 1.5t case ………………..
    win10 says convert to GPT to use full disk
    after you comply windows says ……all ok
    however when you try to copy data to the disk
    it baulks when the 2.2t limit is reached
    interesting….hence my speculation above

    Robbie

    • lui_gough says:

      Yes, it’s all in the bridge chips. By amalgamating 8×512 byte “emulated” sectors as a single 4k “native” sector, they can be partitioned under MBR as they have less than 2^32 sectors. However, if you try to format such a drive even with such a bridge chip, Windows may still insist you use GPT solely based on “capacity” based rules. You can probably get around this with third party partitioning software, or using Linux perhaps.

      Older bridge chips without translation can balk at using the newer drives >2.2Tb. Sometimes they don’t recognize at all because the capacity exceeds their limitations, and other times while they can report the capacity, they can have trouble accessing the drive and can cause corruption because it can “wrap around” and instead start overwriting the system tracks and corrupt the drive entirely. It really depends on the firmware and bridge chipset itself.

      – Gough

      • rob daw says:

        thanks lui
        in other words don’t push your luck
        unless you know how much you know
        and/or you have spent money on the latest partitioning software
        and then you may still not defeat the system/hardware/firmware etc.

        good advice

        robbie

  11. tomman says:

    As a frequent user of cheap USB-PATA/SATA bridges, here are my notes on dealing with AF/large drives:

    – All of my adapters are JMicron-based (JM20337 – combo PATA/SATA, infamously known due to a reference circuit design flaw which can cause data corruption: https://bigacid.wordpress.com/2008/12/08/jm20337-read-data-corruption-solution/ – fortunately mine either have the workaround, or are of a higher revision with the flaw patched on silicon since I’ve yet to experience a single corrupted bit… and yes, I DO HASHCHECK my backups regularly, thankyouverymuch), aside of a lone Sunplus dongle (SPIF215A).

    – All but one of the JMicron (some noname brand) do the 4K translation when an AF drive is detected… but only if the drive is =2TiB drives correctly.

    – Then there is the Sunplus dongle (again, another noname thing, and this one only does SATA), which also does no translation… neither does support anything larger than 2TiB :/

    – This 4K translation CAN be undone. Say your bridge died, you extract the drive and plug it to a native SATA port, and all you can see is a mangled MBR? Well, all you need is to rebuild the MBR to match the proper drive geometry (converting to GPT is a must if the drive is >=2TiB!). This can done with free tools like Testdisk – I wrote a forum post about that: http://www.hddoracle.com/viewtopic.php?f=117&t=1832 – and yes, the filesystem does not seem to care about the underlying block size, hence that’s why I was able to mount it with no data loss/corruption. All you need to do is to look for lost partitions and rebuild the MBR/GPT accordingly. Hell, you should be able to do so if you plug your drive to a 4K-translating bridge too!

  12. Wow! Just found your website .. Looks I’ll be spending here a couple of afternoons 😉

Error: Comment is Missing!