Important bug in esptool2

Over the weekend I found a bug in the checksum calculation in esptool2. It caused some images to have a bad checksum, which would then not be bootable as rBoot would think they were corrupt.

If you haven’t already please pull the latest source and rebuild it. Or if you are on windows and using an old pre-compiled copy you can get an updated version here.

Santander to track customer location via mobile

I found this interesting little note on the bottom corner of my latest Santander current account statement:

santander customer tracking by mobile and tablet

Protecting your account

With effect from 1 July 2015, to prevent and detect fraud, where we hold information about devices you use such as mobiles or tablets, we may use location or other data from these devices. For example, we may check if you’re in the country where your payments are being made in instances where we suspect fraud on your account. We will not use this information for any other purpose.

The simple example given sounds quite reasonable on the surface, as do most surveillance ideas, but:

  • Do we really need to be tracked everywhere we go by another party?
  • What “other data” are they planning on collecting, besides location?
  • What safeguards are there to protect this data?
  • How long will they keep this data?
  • How are they collecting this data? Do you need to have installed their app or will they be getting information from your mobile operator?
  • Who might they be forced to give this data to?
  • Will it just make another massive database that government agencies can access to get more information about persons of interest? I.e. does it just become another, privately operated, extension of government surveillance?
  • Where is the option to opt out?
  • Is it really even needed for the stated purpose? If they have your mobile number and spot a transaction they think is potentially fraudulent they currently call you to confirm it’s really you, which seems to work well when it’s happened to me.

ESP8266 Cache_Read_Enable

Since I haven’t seen this documented anywhere, here is my attempt to explain the Cache_Read_Enable function. Valid values and what they do (at a register level) are from decompiling the code. The outcome of those values is based on my own experimentation so my descriptions and explanations may be silly but they currently fit the observed results.

void Cache_Read_Enable(uint8 odd_even, uint8 mb_count, unt8 no_idea);

Valid values for odd_even:

  • 0 – clears bits 24 & 25 of control register 0x3FF0000C
  • 1 – clears bit 24, sets bit 25
  • other – clears bit 25, sets bit 24

Function of odd_even:

  • 0 – allows access to even numbered mb
  • 1 – allow access to odd numbered mb
  • other – appears to do the same as 1, there must be a difference but I haven’t worked out what it it

Valid values for mb_count:

  • 0-7 – set bits 16, 17 & 18 of control register 0x3FF0000C

Function of mb_count:

  • Which odd or even bank to map (according to odd_even option)
  • e.g. mb_count = 0, odd_even = 0 -> map first 8Mbit of flash
  • e.g. mb_count = 0, odd_even = 1 -> map second 8Mbit of flash
  • e.g. mb_count = 1, odd_even = 0 -> map third 8Mbit of flash
  • e.g. mb_count = 1, odd_even = 1 -> map fourth 8Mbit of flash

Valid values for no_idea:

  • 0 – sets bit 3 of 0x3FF00024
  • 1 – sets bit 26 of 0x3FF0000C and sets bits 3 & 4 of 0x3FF00024

Function of no_idea:
The clue is in the name, I can’t work out what this does from my experiments, but the SDK always sets this to 1.

Memory map limitation – workaround

I was a little hasty in my judgement that this problem could not be solved. While it still appears to be true that only 1MB can be mapped at a time, it is possible to choose which 1MB is mapped. How the mapping was performed was a mystery, to me at least, and I can’t find any info about it on the internet. It was obvious that it was performed by the SDK code, but I hadn’t worked out where.

Since I released rBoot, Espressif have released a new SDK with a new version of the boot loader. This allows you to have two 1MB roms, which is clearly working around the limitation. The nice people at Espressif obviously know the internals of the hardware and have documentation for it, so they can do things that the rest of us wouldn’t know how to (or even know if it was possible).

How does the SDK memory map the flash?

Decompiling the new version of the SDK and comparing it to an older version made it easier to find where the magic happens. The function is Cache_Read_Enable (not well named!) and does not appear to be documented anywhere on the internet. I’ve decompiled it and so I know what the function does, but it communicates with other hardware through memory mapped I/O. Without documentation for that hardware it not easy to really know what it going on beyond this function. As a result some trial and error was required.

The SDK uses new flash size options in the flash header to indicate flash layout as well as size. This method is limited to what the SDK supports and isn’t in a place you want to be rewriting when the config changes (e.g. on an OTA update). So how can rBoot replicate, and improve on, this functionality? Cache_Read_Enable is called from several places in the SDK, because the flash has to be unmapped before normal SPI reads and writes can take place. The SDK SPI read, write and various other functions handle this unmapping (and remapping afterwards) for you. These functions in older versions of the SDK called Cache_Read_Enable directly, but now they all call a wrapper method called Cache_Read_Enable_New, which handles the extra logic involved with rom selection. This gives us a single point which, if we can replace it, would allow us to control the mapping ourselves.

Replacing Cache_Read_Enable_New

So how do we replace it? I first tried using the gcc -wrap option, but it didn’t work. Most references to Cache_Read_Enable_New where replaced with my own code, except those in user_init. It seems that -wrap doesn’t work well when the function you are wrapping is called within the same compilation unit (.o file). Another option is to just define a new method of the same name to override the original. Normally having two matching methods would cause an error at link time, to avoid this we mark the original as ‘weak’ to allow it to be overridden. This isn’t quite as neat, because it requires a small modification to the original libmain.a, but it works!

So, after writing a suitable replacement for Cache_Read_Enable_New I have a working solution. It’s a pity we still can’t map more than 8Mbit at a time, but at least we can now use the whole of larger flash chips in chunks. The new code is now on GitHub. See the readme file for explanation of how to use big flash support.