C Version of Cache_Read_Enable for ESP8266

Just a quick post with a C version of the decompiled ESP8266 rom function Cache_Read_Enable. This function is responsible for memory mapping the SPI flash. I’ve previously discussed it, but a couple of people have wanted the code so it seemed worth posting here. This compiles to quite a few bytes which must stay in iram so, if you want to tamper with the parameters (like rBoot big flash support does), it’s best to write a wrapper to the original rom function rather than use this code to replace it.

void Cache_Read_Enable(uint8 odd_even, uint8 mb_count, uint8 no_idea) {
	
	uint32 base1 = 0x3FEFFE00;
	volatile uint32 *r20c = (uint32*)(base1 + 0x20c);
	volatile uint32 *r224 = (uint32*)(base1 + 0x224);
	
	uint32 base2 = 0x60000200;
	volatile uint32 *r008 = (uint32*)(base2 + 8);
	
	while (*r20c & 0x100) {
		*r20c &= 0xeff;
	}

	*r008 &= 0xFFFDFFFF;
	*r20c &= 0x7e;
	*r20c |= 0x1;
	
	while ((*r20c & 0x2) == 0) {
	}

	*r20c &= 0x7e;
	*r008 |= 0x20000;

	if (odd_even == 0) {
		*r20c &= 0xFCFFFFFF;  // clear bits 24 & 25
	} else if (odd_even == 1) {
		*r20c &= 0xFEFFFFFF;  // clear bit 24
		*r20c |= 0x2000000;   // set bit 25
	} else {
		*r20c &= 0xFDFFFFFF;  // clear bit 25
		*r20c |= 0x1000000;   // set bit 24
	}

	*r20c &= 0xFBF8FFFF; // clear bits 16, 17, 18, 26
	*r20c |= ((no_idea << 0x1a) | (mb_count << 0x10)); // set bits 26 & 18/17/16
	// no_idea should be 0-1 (1 bit), mb_count 0-7 (3 bits)

	if (no_idea == 0) {
		*r224 |= 0x08; // set bit 3
	} else {
		*r224 |= 0x18; // set bits 3 & 4
	}
	
	while ((*r20c & 0x100) == 0) {
		*r20c |= 0x100;
	}

	return;
}

Accessing byte data stored on flash

As ram is short and rom is plentiful on the ESP8266 you might want to store some of your data in rom. The performance won’t be as good as keeping it in ram, but if you need to store a lot of data you may not have any choice. The problem you’ll find is that memory mapped flash needs to be accessed in 4 byte aligned chunks. So as Pete discovered if you try to store a uint8 array in rom you won’t be able to read it in the normal way.

Here is a simple example of how to store and access a uint8 (and uint16) array in rom:


uint8 ICACHE_RODATA_ATTR data[] = {
	0,1,2,3,4,5,6,7,
	8,9,10,11,12,13,14,15
};

uint16 ICACHE_RODATA_ATTR data16[] = {
	300,600,900,1200
};

uint8 ICACHE_FLASH_ATTR read_rom_uint8(const uint8* addr){
	uint32 bytes;
	bytes = *(uint32*)((uint32)addr & ~3);
	return ((uint8*)&bytes)[(uint32)addr & 3];
}

uint16 read_rom_uint16(const uint16* addr){
	uint32 bytes;
	bytes = *(uint32*)((uint32)addr & ~3);
	return ((uint16*)&bytes)[((uint32)addr >> 1) & 1];
}

void ICACHE_FLASH_ATTR user_init(void) {
	os_printf("%d\r\n", read_rom_uint8(&data[0]));
	os_printf("%d\r\n", read_rom_uint8(data + 13));

	os_printf("%d\r\n", read_rom_uint16(&data16[1]));
}

Important bug in esptool2

Over the weekend I found a bug in the checksum calculation in esptool2. It caused some images to have a bad checksum, which would then not be bootable as rBoot would think they were corrupt.

If you haven’t already please pull the latest source and rebuild it. Or if you are on windows and using an old pre-compiled copy you can get an updated version here.

Santander to track customer location via mobile

I found this interesting little note on the bottom corner of my latest Santander current account statement:

santander customer tracking by mobile and tablet

Protecting your account

With effect from 1 July 2015, to prevent and detect fraud, where we hold information about devices you use such as mobiles or tablets, we may use location or other data from these devices. For example, we may check if you’re in the country where your payments are being made in instances where we suspect fraud on your account. We will not use this information for any other purpose.

The simple example given sounds quite reasonable on the surface, as do most surveillance ideas, but:

  • Do we really need to be tracked everywhere we go by another party?
  • What “other data” are they planning on collecting, besides location?
  • What safeguards are there to protect this data?
  • How long will they keep this data?
  • How are they collecting this data? Do you need to have installed their app or will they be getting information from your mobile operator?
  • Who might they be forced to give this data to?
  • Will it just make another massive database that government agencies can access to get more information about persons of interest? I.e. does it just become another, privately operated, extension of government surveillance?
  • Where is the option to opt out?
  • Is it really even needed for the stated purpose? If they have your mobile number and spot a transaction they think is potentially fraudulent they currently call you to confirm it’s really you, which seems to work well when it’s happened to me.

ESP8266 Cache_Read_Enable

Since I haven’t seen this documented anywhere, here is my attempt to explain the Cache_Read_Enable function. Valid values and what they do (at a register level) are from decompiling the code. The outcome of those values is based on my own experimentation so my descriptions and explanations may be silly but they currently fit the observed results.

void Cache_Read_Enable(uint8 odd_even, uint8 mb_count, unt8 no_idea);

Valid values for odd_even:

  • 0 – clears bits 24 & 25 of control register 0x3FF0000C
  • 1 – clears bit 24, sets bit 25
  • other – clears bit 25, sets bit 24

Function of odd_even:

  • 0 – allows access to even numbered mb
  • 1 – allow access to odd numbered mb
  • other – appears to do the same as 1, there must be a difference but I haven’t worked out what it it

Valid values for mb_count:

  • 0-7 – set bits 16, 17 & 18 of control register 0x3FF0000C

Function of mb_count:

  • Which odd or even bank to map (according to odd_even option)
  • e.g. mb_count = 0, odd_even = 0 -> map first 8Mbit of flash
  • e.g. mb_count = 0, odd_even = 1 -> map second 8Mbit of flash
  • e.g. mb_count = 1, odd_even = 0 -> map third 8Mbit of flash
  • e.g. mb_count = 1, odd_even = 1 -> map fourth 8Mbit of flash

Valid values for no_idea:

  • 0 – sets bit 3 of 0x3FF00024
  • 1 – sets bit 26 of 0x3FF0000C and sets bits 3 & 4 of 0x3FF00024

Function of no_idea:
The clue is in the name, I can’t work out what this does from my experiments, but the SDK always sets this to 1.

Memory map limitation – workaround

I was a little hasty in my judgement that this problem could not be solved. While it still appears to be true that only 1MB can be mapped at a time, it is possible to choose which 1MB is mapped. How the mapping was performed was a mystery, to me at least, and I can’t find any info about it on the internet. It was obvious that it was performed by the SDK code, but I hadn’t worked out where.

Since I released rBoot, Espressif have released a new SDK with a new version of the boot loader. This allows you to have two 1MB roms, which is clearly working around the limitation. The nice people at Espressif obviously know the internals of the hardware and have documentation for it, so they can do things that the rest of us wouldn’t know how to (or even know if it was possible).

How does the SDK memory map the flash?

Decompiling the new version of the SDK and comparing it to an older version made it easier to find where the magic happens. The function is Cache_Read_Enable (not well named!) and does not appear to be documented anywhere on the internet. I’ve decompiled it and so I know what the function does, but it communicates with other hardware through memory mapped I/O. Without documentation for that hardware it not easy to really know what it going on beyond this function. As a result some trial and error was required.

The SDK uses new flash size options in the flash header to indicate flash layout as well as size. This method is limited to what the SDK supports and isn’t in a place you want to be rewriting when the config changes (e.g. on an OTA update). So how can rBoot replicate, and improve on, this functionality? Cache_Read_Enable is called from several places in the SDK, because the flash has to be unmapped before normal SPI reads and writes can take place. The SDK SPI read, write and various other functions handle this unmapping (and remapping afterwards) for you. These functions in older versions of the SDK called Cache_Read_Enable directly, but now they all call a wrapper method called Cache_Read_Enable_New, which handles the extra logic involved with rom selection. This gives us a single point which, if we can replace it, would allow us to control the mapping ourselves.

Replacing Cache_Read_Enable_New

So how do we replace it? I first tried using the gcc -wrap option, but it didn’t work. Most references to Cache_Read_Enable_New where replaced with my own code, except those in user_init. It seems that -wrap doesn’t work well when the function you are wrapping is called within the same compilation unit (.o file). Another option is to just define a new method of the same name to override the original. Normally having two matching methods would cause an error at link time, to avoid this we mark the original as ‘weak’ to allow it to be overridden. This isn’t quite as neat, because it requires a small modification to the original libmain.a, but it works!

So, after writing a suitable replacement for Cache_Read_Enable_New I have a working solution. It’s a pity we still can’t map more than 8Mbit at a time, but at least we can now use the whole of larger flash chips in chunks. The new code is now on GitHub. See the readme file for explanation of how to use big flash support.

Memory map limitation affecting rBoot

I’ve become aware of a serious limitation that testing should should have found, if I’d done a bit more of it! The ESP8266 only memory maps the the first 8 Mbit of the SPI flash. rBoot doesn’t use memory mapped flash, it uses SPI reads, so rBoot itself is fine with any size of flash. The problem comes if you try to put an irom section beyond 8 MBit (i.e. from address 0x40300000) – code there won’t be accessible at run time!

I’m quite disappointed about that. It’s something that rBoot can’t overcome, it’s a limitation of the ESP8266 design. You can still use the extra space on larger chips for something, like logging data, or for storing a filesystem for your resources. You can also still use rBoot OTA to flash these resources by dedicating a non-bootable rom slot to them. Your code would need to access them with SPI reads rather than through the memory mapping though.

Update: this problem has been largely mitigated, see here.

esptool2 – a rom creation tool for ESP8266

Time to write something about esptool2, as I’ve been using it in the last few posts. The SDK way of creating rom images is a mess involving a shell script / batch file and Makefile. I did find a windows version of esptool by mamalala, but it didn’t create the newer rom format used by the SDK boot loader. So I wrote my own. I used a simpler design and command line compared to the version by mamalala (which I think is designed to be a drop in replacement with compatible command line interface). Although I prefer it this way I didn’t call it esptool2 to imply it was better or a replacement and I hadn’t originally intended to release it.

Features

  • Create old style rom images, e.g. eagle.flash.bin & eagle.irom0text.bin (for use without a boot loader).
  • Create rom images for the first type SDK boot loader.
  • Create rom images for the newer (v1.2+) SDK boot loader.
  • Create rom images for rBoot (because it uses the same format as as SDK v1.2 roms).
  • Export elf sections as bytes in a C header file (used when building rBoot itself).
  • Single output file per command, perfect for use in a Makefile.
  • Open source.
  • Compiles under Visual Studio on Windows and GCC on Linux.

It doesn’t flash though, so you’ll need something else to do that, sorry. Never done any serial programming in C and I suspect it’s probably difficult to do in a nice platform independent way. If you can do this please get in touch.

Get the source from github: https://github.com/raburton/esptool2 Or get the compiled version for windows: https://www.dropbox.com/s/03ci9orf9ccr9ya/esptool2.exe?dl=0

Run it to see full usage instructions or look at the rBoot sample project for usage examples.

rBoot tutorial for ESP8266 – OTA updates

Ok, hope you’re still with me after my previous massive post. Now I’m going to show you how to perform an over-the-air (OTA) update with rBoot. I’ve covered all the background already, so this should be pretty straight forward as long as you have a simple two rom rBoot setup running.

Now add rboot.h, rboot-ota.h and rboot-ota.c to your project and call the rboot_ota_start function. How you invoke the OTA update code is up to you, the sample project on GitHub (now updated) has a simple command line interface over the UART allowing the user to enter the command “ota”.

rboot_ota_start takes an rboot_ota struct with the options for the update.

typedef struct {
	uint8 ip[4];
	uint16 port;
	uint8 *request;
	uint8 rom_slot;
	ota_callback callback;
} rboot_ota;
  • ip is the ip address of the web server to download the new rom from.
  • port is the web server port (usually 80).
  • request is a complete http request which will be sent to the web server, this may seem like a slightly odd way to do it, but it gives you full control over what is sent and it’s the same way the SDK OTA update works.
  • rom_slot is the number of the rom slot on the flash to update, starting at zero. In our two rom example that will be either 0 or 1 (and the opposite to the one we are currently running).
  • callback is a user function that will be called when the update is completed (either success or failure), it is passed a pointer to the rboot_ota structure and a bool to indicate success or failure. This is where you will then switch to the new rom (using rboot_set_current_rom) and restart the device.

Example

static void ICACHE_FLASH_ATTR OtaUpdate_CallBack(void *arg, bool result) {

	char msg[40];
	rboot_ota *upServer = (rboot_ota*)arg;

	if(result == true) {
		// success, reboot
		os_sprintf(msg, "Firmware updated, rebooting to rom %d...\r\n", upServer->rom_slot);
		uart0_send(msg);
		rboot_set_current_rom(upServer->rom_slot);
		system_restart();
	} else {
		// fail, cleanup
		uart0_send("Firmware update failed!\r\n");
		os_free(upServer->request);
		os_free(upServer);
	}
}

static const uint8 ota_ip[] = {192,168,7,5};
#define HTTP_HEADER "Connection: keep-alive\r\nCache-Control: no-cache\r\nUser-Agent: rBoot-Sample/1.0\r\nAccept: */*\r\n\r\n"

static void ICACHE_FLASH_ATTR OtaUpdate() {

	uint8 slot;
	rboot_ota *ota;

	// create the update structure
	ota = (rboot_ota*)os_zalloc(sizeof(rboot_ota));
	os_memcpy(ota->ip, ota_ip, 4);
	ota->port = 80;
	ota->callback = (ota_callback)OtaUpdate_CallBack;
	ota->request = (uint8 *)os_zalloc(512);

	// select rom slot to flash
	slot = rboot_get_current_rom();
	if (slot == 0) slot = 1; else slot = 0;
	ota->rom_slot = slot;

	// actual http request
	os_sprintf((char*)ota->request,
		"GET /%s HTTP/1.1\r\nHost: "IPSTR"\r\n" HTTP_HEADER,
		(slot == 0 ? "rom0.bin" : "rom1.bin"),
		IP2STR(ota->ip));

	// start the upgrade process
	if (rboot_ota_start(ota)) {
		uart0_send("Updating...\r\n");
	} else {
		uart0_send("Updating failed!\r\n\r\n");
		os_free(ota->request);
		os_free(ota);
	}

}

It’s really that simple, that’s all you need to add to your application to be able to perform OTA updates. You might want to put in a version check, so it only updates if there is a new version, but that’s up to you.

Web Server

All that remains is to drop rom0.bin and rom1.bin in the root of your web server. Obviously you can change where it looks for the files with a small tweak to the code above.

A sample rBoot project

To help illustrate my previous explanation of getting started with rBoot I’ve produced a simple sample project. It doesn’t do anything very exciting itself, but it shows you how to compile, link and build roms that will work with rBoot. When I’ve written my next post on OTA updating with rBoot I’ll add an implementation to the project to demonstrate that too.

https://github.com/raburton/rboot-sample

I realise my last post was also very long (actually most of my posts are), so this code should serve as a TL;DR version of the previous post for those who prefer to just get stuck in.

If you wanted to use the SDK boot loader instead of rBoot this project would still serve as a useful example.