Tag Archives: rboot

Memory map limitation affecting rBoot

I’ve become aware of a serious limitation that testing should should have found, if I’d done a bit more of it! The ESP8266 only memory maps the the first 8 Mbit of the SPI flash. rBoot doesn’t use memory mapped flash, it uses SPI reads, so rBoot itself is fine with any size of flash. The problem comes if you try to put an irom section beyond 8 MBit (i.e. from address 0x40300000) – code there won’t be accessible at run time!

I’m quite disappointed about that. It’s something that rBoot can’t overcome, it’s a limitation of the ESP8266 design. You can still use the extra space on larger chips for something, like logging data, or for storing a filesystem for your resources. You can also still use rBoot OTA to flash these resources by dedicating a non-bootable rom slot to them. Your code would need to access them with SPI reads rather than through the memory mapping though.

Update: this problem has been largely mitigated, see here.

esptool2 – a rom creation tool for ESP8266

Time to write something about esptool2, as I’ve been using it in the last few posts. The SDK way of creating rom images is a mess involving a shell script / batch file and Makefile. I did find a windows version of esptool by mamalala, but it didn’t create the newer rom format used by the SDK boot loader. So I wrote my own. I used a simpler design and command line compared to the version by mamalala (which I think is designed to be a drop in replacement with compatible command line interface). Although I prefer it this way I didn’t call it esptool2 to imply it was better or a replacement and I hadn’t originally intended to release it.

Features

  • Create old style rom images, e.g. eagle.flash.bin & eagle.irom0text.bin (for use without a boot loader).
  • Create rom images for the first type SDK boot loader.
  • Create rom images for the newer (v1.2+) SDK boot loader.
  • Create rom images for rBoot (because it uses the same format as as SDK v1.2 roms).
  • Export elf sections as bytes in a C header file (used when building rBoot itself).
  • Single output file per command, perfect for use in a Makefile.
  • Open source.
  • Compiles under Visual Studio on Windows and GCC on Linux.

It doesn’t flash though, so you’ll need something else to do that, sorry. Never done any serial programming in C and I suspect it’s probably difficult to do in a nice platform independent way. If you can do this please get in touch.

Get the source from github: https://github.com/raburton/esptool2 Or get the compiled version for windows: https://www.dropbox.com/s/03ci9orf9ccr9ya/esptool2.exe?dl=0

Run it to see full usage instructions or look at the rBoot sample project for usage examples.

rBoot tutorial for ESP8266 – OTA updates

Ok, hope you’re still with me after my previous massive post. Now I’m going to show you how to perform an over-the-air (OTA) update with rBoot. I’ve covered all the background already, so this should be pretty straight forward as long as you have a simple two rom rBoot setup running.

Now add rboot.h, rboot-ota.h and rboot-ota.c to your project and call the rboot_ota_start function. How you invoke the OTA update code is up to you, the sample project on GitHub (now updated) has a simple command line interface over the UART allowing the user to enter the command “ota”.

rboot_ota_start takes an rboot_ota struct with the options for the update.

typedef struct {
	uint8 ip[4];
	uint16 port;
	uint8 *request;
	uint8 rom_slot;
	ota_callback callback;
} rboot_ota;
  • ip is the ip address of the web server to download the new rom from.
  • port is the web server port (usually 80).
  • request is a complete http request which will be sent to the web server, this may seem like a slightly odd way to do it, but it gives you full control over what is sent and it’s the same way the SDK OTA update works.
  • rom_slot is the number of the rom slot on the flash to update, starting at zero. In our two rom example that will be either 0 or 1 (and the opposite to the one we are currently running).
  • callback is a user function that will be called when the update is completed (either success or failure), it is passed a pointer to the rboot_ota structure and a bool to indicate success or failure. This is where you will then switch to the new rom (using rboot_set_current_rom) and restart the device.

Example

static void ICACHE_FLASH_ATTR OtaUpdate_CallBack(void *arg, bool result) {

	char msg[40];
	rboot_ota *upServer = (rboot_ota*)arg;

	if(result == true) {
		// success, reboot
		os_sprintf(msg, "Firmware updated, rebooting to rom %d...\r\n", upServer->rom_slot);
		uart0_send(msg);
		rboot_set_current_rom(upServer->rom_slot);
		system_restart();
	} else {
		// fail, cleanup
		uart0_send("Firmware update failed!\r\n");
		os_free(upServer->request);
		os_free(upServer);
	}
}

static const uint8 ota_ip[] = {192,168,7,5};
#define HTTP_HEADER "Connection: keep-alive\r\nCache-Control: no-cache\r\nUser-Agent: rBoot-Sample/1.0\r\nAccept: */*\r\n\r\n"

static void ICACHE_FLASH_ATTR OtaUpdate() {

	uint8 slot;
	rboot_ota *ota;

	// create the update structure
	ota = (rboot_ota*)os_zalloc(sizeof(rboot_ota));
	os_memcpy(ota->ip, ota_ip, 4);
	ota->port = 80;
	ota->callback = (ota_callback)OtaUpdate_CallBack;
	ota->request = (uint8 *)os_zalloc(512);

	// select rom slot to flash
	slot = rboot_get_current_rom();
	if (slot == 0) slot = 1; else slot = 0;
	ota->rom_slot = slot;

	// actual http request
	os_sprintf((char*)ota->request,
		"GET /%s HTTP/1.1\r\nHost: "IPSTR"\r\n" HTTP_HEADER,
		(slot == 0 ? "rom0.bin" : "rom1.bin"),
		IP2STR(ota->ip));

	// start the upgrade process
	if (rboot_ota_start(ota)) {
		uart0_send("Updating...\r\n");
	} else {
		uart0_send("Updating failed!\r\n\r\n");
		os_free(ota->request);
		os_free(ota);
	}

}

It’s really that simple, that’s all you need to add to your application to be able to perform OTA updates. You might want to put in a version check, so it only updates if there is a new version, but that’s up to you.

Web Server

All that remains is to drop rom0.bin and rom1.bin in the root of your web server. Obviously you can change where it looks for the files with a small tweak to the code above.

A sample rBoot project

To help illustrate my previous explanation of getting started with rBoot I’ve produced a simple sample project. It doesn’t do anything very exciting itself, but it shows you how to compile, link and build roms that will work with rBoot. When I’ve written my next post on OTA updating with rBoot I’ll add an implementation to the project to demonstrate that too.

https://github.com/raburton/rboot-sample

I realise my last post was also very long (actually most of my posts are), so this code should serve as a TL;DR version of the previous post for those who prefer to just get stuck in.

If you wanted to use the SDK boot loader instead of rBoot this project would still serve as a useful example.

A boot loader tutorial for ESP8266 using rBoot

From recent reading and discussions it seems that lots of people aren’t using the Espressif boot loader (or my own rBoot, but that’s less of a surprise) on the ESP8266. Why? Maybe people aren’t aware of the reasons why you might want to. Or maybe they can’t figure out how to – when I started playing with the boot loader it was poorly documented (probably still is) and I had to work it out myself.

I talk about “rom” or “roms” here to mean full compiled user apps, that might traditionally have been deployed on their own but with a boot loader you can have several on the flash with just one operating at a time. To avoid confusion (hopefully) I’ll refer to the rom section of code (the code that is run from rom, usually just the .irom0.text elf section) as irom from now on.

Why use a boot loader?

The main reason is to allow you to have multiple “roms” and to be able to switch between them. That may not sound quite as useful as being able to dual boot your computer between Windows and Linux, but there are uses for it. For example, if you want to update your device over the air (OTA) you’ll need to have at least two rom slots on your flash, a running one and one that’s getting flashed with the new version (which you then reboot into). There are work-arounds you could do to OTA update a device from the running rom, but it wouldn’t be very safe. OTA updates are probably the main reason for wanting to use a boot loader. However, you might have a need to deploy a device with two completely different functions and not want to combine them into a single application. With a boot loader you could put both separate apps on the device and switch between them remotely or with a GPIO etc.

Another reason is that the boot loader can load your application differently to the built in loader and add features not present in the original loader. For example the built in loader checks a checksum for the elf sections it loads into iram, but not for the .irom0.text section that is run from rom (this is often referred to as the SDK lib, but that’s also where your code goes if you mark it with ICACHE_FLASH_ATTR). A boot loader could add this extra check (rBoot can, it’s coded but not currently pushed to git), the Espressif loader doesn’t).

The Espressif 1.2 loader introduced a new format for the roms that puts the irom section (aka SDK lib) first and the iram sections immediately after. This means you don’t need to work with the arbitrary split of iram going before 0x40000 and irom after (you can change this split, but you may need to keep changing it as your sections change). Now the iram sections are still limited in size (because there is a finite amount of iram) but the irom section can be pretty much as large as the flash chip (minus the space needed for the iram sections, boot loader and SDK config (last 4 sectors of flash)), without needing to play with the linker scripts. With rBoot your options for laying out the flash are unlimited, so at present I’m not supplying sample linker files, but I do explain how to make them (it’s not hard).

How do you use the boot loader?

1) Think about how you want to lay out your flash, particularly how many roms you want and if they should be the same sizes. See below for a worked example.
2) Compile your code in the normal way.
3) Link your code slightly differently. For each rom slot on the flash you need a linker script – a copy of the standard eagle.app.v6.ld with one simple change. You need to link your object files against each one, to produce multiple elf files. See below for examples and an explanation.
4) Use esptool to build a rom image from each of the linked elf files, this time using the ‘boot_v1.2+’ option.
5) Write the boot loader to 0x00000 on the flash.
6) If you want something more than the simple 2 rom default create and write a boot loader config sector to the flash at 0x01000.
6) Write the roms to the flash at the appropriate addresses (see below).
7) Enjoy.

Linker Files

Why do you need new linker files and why do you need to link several times? I’ll assume if you are programming in C you understand the concepts of compiling and linking (but if you don’t it doesn’t really matter). The boot loader copies most sections to iram, this means they will always end up where you want them to be in memory, regardless of where they get placed on the flash. However the irom sections (usually just .irom0.text) aren’t copied, access to that code is via the memory mapped SPI flash. The whole chip is mapped at a base address of 0x40200000 so when you have multiple roms on your flash the irom section for each rom will be at a different place on the flash and so a different place in memory. When the code is linked the linker needs to know where that code will be in memory and the way to tell it is via the linker script. Short version: each copy of the rom on the flash needs to have been linked differently depending on where it will be flashed.

Example

I’ll make this one easy – a two rom setup, both the same size. A simple but common scenario (all you would need for an app with OTA updates) and rBoot will self configure for this when you run it.

We want two roms on the flash, so the sensible thing to do is place one at the beginning and the other half way along. We need to leave space for the boot loader and the config so we can’t put the first one at the very beginning, so we’ll start it at sector 3, flash address 0x02000. There is no reason that the second can’t start exactly half way into the rom, but for symmetry we’ll start that at half+0x02000 (and rBoot’s default config expects this). Lets say you have an ESP8266 board with an 4Mbit SPI flash, that second rom is going to be written to 0x42000, for 8MBit it will be 0x82000. If you have a flash larger than 8Mb the default position for the second rom will remain at 0x82000 due to the memory mapping limitation.

Now we need to make two appropriate linker files. Copy eagle.app.v6.ld to rom0.ld and rom1.ld. Edit rom0.ld and change the value of ‘irom0_0_seg org = ‘ from 0x40240000 to 0x40202010. This is the base memory mapped flash address (0x40200000) + our chosen flash address (0x02000) + the length of an extra header (0x10). Edit rom1.ld and set the value to 0x40282010. You can increase the len parameter in both of these too if you need more space for your irom section. Just don’t make it so big that it will overflow into the flash space of the next rom (or push the iram sections into it) i.e. the rom must come out at <512KB if you are going to fit two of them on a 1MB flash.

Now edit your Makefile and find the linker line, which is likely to start $(LD). You need to duplicate this and make one of them use rom0.ld, instead of eagle.app.v6.ld, and the other to use rom1.ld. Also make sure they output two different files, don’t have the second one write over the output of the first!

Now you should have two elf files, previously you would have just had one. What normally happens next appears to involve black magic, a Makefile, a shell script/batch file (gen_misc) and a python script (gen_appbin). This produces the flashable rom from the elf file. The whole thing is a mess and can be greatly simplified by using my esptool2 to build roms (other simplified tools also exist, I called mine esptool2 not to imply it is better or supersedes other tools, but just to distinguish it on my own system, way before I intended to release it). The key thing here is that you need to run it twice now, to produce two rom files from the two different elf files. You also need to instruct it to produce roms for what it describes as “boot_v1.2+” (the SDK boot loader v1.2+). However you are currently calling this will need to be updated, but I’d suggest switching to esptool2 if you’re using the original SDK code to do this.

Now just flash the three files (two roms and rBoot itself):
e.g. esptool.py –port COM7 -fs 8m 0x00000 rboot.bin 0x02000 rom0.bin 0x82000 rom1.bin

Using ‘-fs 8m’ here is important, it ensures the flash size is stored in the first few bytes of the flash, this will be read by rBoot to determine the flash size so it can work out where the half way point is.

On first boot you’ll see a message that a default config is being created and all being well the first rom will start. Assuming you got this far, hold on for part two where I’ll show you how to switch rom and/or OTA update from your app…

rBoot – A new boot loader for ESP8266

As promised here is my new boot loader for the ESP8266 – rBoot.

Advantages over SDK supplied bootloader:

  • Open source (written in C) – this is the big one.
  • Supports any number of roms.
  • Roms can be different sizes.
  • Rom slots can be used for resource storage as well as bootable apps (and benefit from the OTA update system).
  • Can use the full size of the SPI flash (see below).
  • Rom slots can be altered after deployment (with care!).
  • Earlier rom validation (less prone to errors).
  • Can try multiple backup roms (without needing to reboot).
  • Rom selection by GPIO (e.g. hold down a button when powering on to start a recovery rom).
  • Wastes no stack space  (SDK boot loader uses 144 bytes).
  • Documented config structure (easy to configure from user code).

Disadvantages over SDK supplied bootloader:

  • Not compatible with sdk libupgrade (but equivalent source included, based on open source copy shipped with earlier SDKs, so you can easily update your existing OTA app use this new code).
  • Requires you to think slightly more about your linker scripts, rather than just using the pair supplied with the SDK (but it’s not really that difficult – if you’re programming in C it’ll be well within your capabilities).

Problems common to both:

  • You still need to relink user code against multiple different linker scripts depending where you intend to place it on the flash, because the memory mapped position of the .irom0.text section needs to be known at link time. This also prevents you moving roms around at will once they have been compiled.
  • Only 8MBit of flash can be memory mapped at a time (the SDK bootloader allows at most the first 2 x 8Mbit chunks to be used for roms, rBoot doesn’t have this limit, on a 32MBit flash you can have 4 x 8MBit roms), see memory mapping imitation for more details.

Source code

I’ve decided to start putting my source code on GitHub, it’ll be easier to maintain keep my blog tidier.

https://github.com/raburton