Ledger suffered a supply-chain attack which exposes, as attacks are wont to do, trust dependencies people formerly ignored. In this case the attack was against the physical supply chain with someone, so far unknown, planting surveillance hardware inside Ledgers sold around the world. Supply chain attacks against software are not new. They are in fact quite common. And as a Ledger is a small computer you carry around it is clearly open to attacks against both its physical and software supply chains.
Physical supply chain attacks are not new. Bloomberg ran a detailed and controversial piece about allegations of long-term physical supply chain attacks in the computer space in 2021. Spies planting bugs is a staple of movies forever. The question is what we can learn from this recent attack. And what can people do to protect themselves. We have never thought Ledger-like fancy signing devices were a good idea. And explaining why provides much of the background needed to help avoid falling victim to these kinds of attacks.
Clearly the best and most secure signing system is one in which the private keys are never connected to the internet and all information exchanged from the signing process to the internet is 100% guaranteed, physically blocked, one way only. Conceptually you can think of this as doing the signing math by hand and burning the scrap paper. Or keeping your private keys on a computer with zero internet connectivity and conveying signed messages using writable DVDs. Why DVDs? Because you can put a read-write device in your offline computer and a read-only device in your online computer and be 100% sure no information flows the wrong way.
Use Cases
That DVD-based process is cumbersome and slow. It is actually not particularly error-prone if you have the right physical setup (i.e. offline laptop is physically incapable of an internet connection and read-only vs read-write configured as above). But it is not easily portable and is not very convenient. Whatever information you need to sign needs to be securely moved from the internet to your offline computer. Doing that manually is painful. Doing that with digital devices is risky.
Enter devices like a Ledger. These devices are all about simplifying the workflow. Simple is good. And complex is bad. Signing devices with Bluetooth and WiFi are ridiculous. These are often ridiculed and with good reason. But users need a way to interact with their devices and that itself is risky. Unless you do everything with pen and paper you have some kind of dependence on software.
Maybe you think a tiny keyboard on the device as the only input method is a good idea. What if the keyboard breaks? Offering a fallback scheme can be risky. Not offering a fallback scheme means you need two devices. Which have to be secured and kept synchronized. You can dream up scenarios that must be managed fairly easily. All of this is relevant when losing access means losing everything.
The key is to understand what you are trying to do and then trying to do that thing and that thing alone. Why? Because narrowing the task narrows the set of compromises you must inevitably chose among.
Bad Compromises
The goal of a signing device is to simply the workflow so there is less risk. But you need to view this as a whole. Simplifying the user workflow but introducing Bluetooth and WiFi into the process is a bad compromise. Requiring passwords so long and complex people are going to store them insecurely is a bad compromise. Using tamper resistant hardware that is so fragile users are going to need half a dozen devices is a bad idea. Each defense must be considered alongside any problems it introduces.
All of the possible compromises live in some large design space. And designing a good device is challenging. You need to understand your target users and what they are capable of and what they want. You also need to understand their tolerance for inconvenience. The process described above using DVDs and a computer with zero internet connectivity is not a good design for a group of users that think disabling WiFi in Windows is what we mean by "physically incapable of an internet connection."
Some of this is on the user of course. Products should lay out how they work and what they do – and users should be honest with themselves about whether they understand that information well enough to use the product securely. Companies should really do a better job telling potential users what it takes to be a good user and discouraging people that do not fit from buying their product. This extends well outside web3 of course. And maybe it is unrealistic to expect companies to forgo revenue from clueless buyers. But that, too, is a risk users should know about.
This is all about avoiding bad compromises. Ledger itself suffered a supply chain attack against their software in 2023 which depended on the ability of users to upgrade the software on their Ledger. Supply chain attacks on software are attempts to introduce vulnerabilities in software packages that are used as components within some larger downstream product. The idea is that you add a subtle vulnerability in package X, product Y uses package X somewhere, and then once everybody upgrades their version of Y to include the bad X you are able to exploit the vulnerability.
This only works if users can change the software on the Ledger. Introducing a software upgrade feature is itself a compromise. Yes it means known security problems can be patched. And it means new features can be added. But it also means supply chain attacks become possible. These are challenging compromises. Imagine the software on a hardware wallet was fixed when assembled and could never be changed. Then the attack would not have worked against existing devices . But if the software is static users need to buy new devices over time to support new blockchains and products and such. They need to buy new devices to get bug fixes. More devices. More shuffling of signatures around. Users depending on multiple versions of things and with assets more spread out. Is that better or worse? It is not at all obvious.
So while Bluetooth is pretty obviously a terrible idea lots of other compromises are far more grey.
Complexity
The other main challenge is complexity. The harder it is to do something correctly the greater the chance users do it incorrectly. The more complex a process the more likely there is a vulnerability you did not notice.
Strictly speaking mistakes are the user's fault. if they follow a process wrong it is their problem. And the legal department at every signing device manufacturer surely holds this position. But for a designer designing a product, or a user considering what product to buy, the reality is different. You want to design and buy products that are easier to use safely than to use wrong. This kind of thinking is central in fields like car design. Modern cars make lots of noise if you try to set off without seatbelts. Modern convertibles will not let you move the roof up or down if you are travelling above some set maximum speed (often 0!) lest you rip the roof off or damage the mechanism. Yes adding sensors makes these systems more complex. And yes those sensors can fail which, overall, means maintenance efforts go up. But the overall designs make things safer and better. Cars are surely far safer than they were decades ago.
Same with airbags. Airbags are entirely passive devices from the driver's perspective. But there are all kinds of sensors involved in running the airbags. Airbags do, very occasionally, deploy when they should not. Complexity is never good in and of itself – but it can be part of a better design if properly considered.
But a lot of these signing devices are quite complex computers with awful failure modes (i.e. all your money disappears) that do not look like they were designed with overall reliability in mind. For a moment think about using a generic laptop as your signing device. What would it take to compromise the laptop supply chain? Well there are a huge number of manufacturers making a huge number of models. These models change all the time. And only a tiny % of laptops are going to end up being used as signing devices. Compromising one particular model may or may not be difficult. Compromising them all is impossible.
The tiny % that will be used for signing is incredibly important because an attacker will compromise every device in their target zone but only get something back from a vanishingly small % of compromised devices. Whatever it costs to exploit a single device, it is surely more expensive to run that across a mass-market product with huge distribution than a niche market product like a Ledger.
This column was written on a Lenovo ThinkPad T16 Gen 1 from 2022. Say someone managed to compromise that particular model during manufacturing. Lenovo makes dozens of models. Lenovo is one of many manufacturers. And that model was made for about one year. Taking the risk to get in the supply chain and compromise that model is maybe not worth it for the tiny number of devices you would get to control. The vast majority of those devices do boring work inside companies. All the money spent compromising those devices is a total writeoff.
Complexity is a factor here too. Maybe you want to compromise all of Lenovo (or some other manufacturer). Or do this year after year. The chances you get caught go up and up and up. Recompromising new designs every year is extremely risky and likely unsustainable.
But 100% of Ledgers are used as signing devices. There are not many models. And when you compromise a Ledger you get all the assets secured by that device. This is great target. Given the nature of the device, the attack is easier the more complex the assembly and distribution processes are because more complex processes yield more points to attack . In this direction complexity is the attackers friend. And something like laptops, which at first appear more complex and more vulnerable, are instead by their very nature harder for attackers to exploit usefully. Remember criminals care about risk/reward too.
For this reason Ledger should have always had better tamper protection and provided better schematics to verify devices. If you have spent any time buying random cheap electronics in sketchy stores around the world you have seen components inside that were very obviously not from the brand named on the outside of the box. Anyone that has done any soldering work can spot these problems. Maybe that is too much to expect from hardware wallet users. But hardware wallet users can count and comparing the number of chips inside a device against a schematic is not too much to expect.
To the extent none of the major hardware wallet vendors do this well: you need to pick from what is on the menu. If only bad hardware wallets are available then maybe some other solution is better for you. Recognize what you are actually buying and using and trusting. And if nobody sells a product that suits the workflow you want it is on you to design a new workflow that is feasible with the things you can buy .
Reality
This Ledger attack seems to have involved adding a few small bits of hardware inside the case. Ledgers are not shipped like precious gems. Nor, for that matter, are other electronics. Getting into the supply chain and modifying devices for many types of electronics is not that hard. Remember we are talking about criminals here and they manage to smuggle drugs. Measured against other activities we know happen out there in the world this Ledger attack seems perfectly plausible.
Ledgers, and devices like them, are at more risk simply because the expected payoff for each compromise is far higher. This is true even if you expect well over 50% of the exploited devices to pay off nothing because the fraction of exploited hardware wallets that pay off is still going to be wildly higher than that for any other class of consumer electronics.
Hardware wallets are not, in general, distributed in tamper-resistant packaging with tamper-resistant seals as are found on essentially all the mobile phones and laptops out in the world. One thing to note here is that phones and laptops do not have seals primarily for user protection – they have seals to help the company reject warranty claims from users that break their own devices. Hardware wallet designers should do a better job building in tamper-resistance because having the right hardware and only the right hardware is an essential feature of the product. When Apple uses weird screws and places seals inside a phone they are trying to protect Apple. Users do not care. But hardware wallet manufacturers can use these features as selling points because users should, and now probably do, care.
Users should also consider whether moving to a simple thumb drive and offline laptop is better for them. While it is not as foolproof as the setup described above, you can get a long way with thumb drives and a laptop with the internet disabled. Does that have risks? Sure. But there is essentially no chance anyone tampered with a $1 1GB thumb drive because it would be too expensive. Certainly the risk profile is different from a hardware wallet. Diversification and all that.
And in general users need to take more responsibility on themselves. It is called self-custody after all. Why would you depend on a small box without a tamper seal and without published schematics to check your device is correct? Yes Ledger has a feature to determine if the secure element inside is correct. This is essentially worthless as we just saw because it confirms one component of the software +hardware is correct but says nothing about spyware that might exist around it. That is yet another example of web3 engineers misrepresenting what trusted computing is and is not. But that is a column for another day.
The real question here is whether hardware wallet users want security or security theater. Blindly trusting a $100 box you got in the mail that was manufactured in some major electronics hub and shipped to your door is a strange substitute for trusting a centralized custodian. A hardware wallet with WiFi and Bluetooth is security theater. Does that mean you need to go through the firmware line-by-line on every device you use? Be realistic. We are not claiming any and all trust on anything is stupid. But different dependencies have different risk profiles.
If you are going to self-custody you need to understand what you are trusting and mitigate where you can. In this most recent Ledger attack plenty of people trusted non-sealed boxes they picked up somewhere for about $100 which had sloppy amateur wiring inside and visible antennas that were a clear red flag. We are absoltely not saying these problems would be obvious to everyone or that this makes it all the users own fault. But it should be far easier for users to verify things are right than it is. And wallet providers really should not provide low-value assurances around the secure element that are likely to confuse users and effectively trick them into trusting unverified electronics.
