NodeMCU Reference Manual¶
NodeMCU firmware is an IoT project ("the Project") which implements a Lua-based runtime for SoC modules based on the Espressif ESP8266 and ESP32 architectures. This NodeMCU Reference Manual (NRM) specifically addresses how the NodeMCU Lua implementation relates to standard Lua as described in the two versions of the Lua language that we currently support:
Developers using the NodeMCU environment should familiarise themselves with the 5.3 LRM.
The Project provides a wide range of standard library modules written in both C and Lua to support many ESP hardware modules and chips, and these are documented in separate sections in our online documentation.
The NRM supplements LRM content and module documentation by focusing on a complete description of the differences between NodeMCU Lua and standard Lua 5.3 in use. It adopts the same structure and style as the LRM. As NodeMCU provides a full implementation of the Lua language there is little content herein relating to Lua itself. However, what NodeMCU does is to offer a number of enhancements that enable resources to be allocated in constant program memory — resources in standard Lua that are allocated in RAM; where this does impact is in the coding of C library modules and the APIs used to do this. Hence the bulk of the differences relate to these APIs.
One of our goals in introducing Lua 5.3 support was to maintain the continuity of our existing C modules by ensuring that they can be successfully compiled and executed in both the Lua 5.1 and 5.3 environments. This goal was achieved by a combination of: - enabling relevant compatibility options for standard Lua libraries; - back porting some Lua 5.3 API enhancements back into our Lua 5.1 implementation, and - making some small changes to the module source to ensure that incompatible API use is avoided.
Further details are given in the Lua compatibility section below. Notwithstanding this, the Project has now deprecated Lua 5.1 and will soon be moving this version into frozen support.
As well as providing the ability to building runtime firmware environments for ESP chipsets, the Project also offers a
luac.cross cross compiler that can be built for common platforms such as Windows 10 and Linux, and this enables developers to compile source modules into a binary file format for download to ESP targets and loading from those targets.
The NodeNCU runtime offers a full implementation of the LRM §2 core concepts, with the following adjustments:
Values and Types¶
- The firmware is compiled to use 32-bit integers and single-precision (32-bit) floats. Address pointers are also 32-bit, and this allows all Lua variables to be encoded in RAM as an 8-byte "
TValue" (compared to the 12-byte
TValueused in Lua 5.1).
- C modules can statically declare read-only variant of the
Tablestructure type known as a "
ROTable". There are some limitations to the types for ROTable keys and value, in order to ensure that these are consistent with static declaration. ROTables are stored in code space (in flash memory on the ESPs), and hence do not take up RAM resources. However these are still represented by the Lua type table and a Lua application can treat them during execution the same as any other read-only table. Any attempt to write to a
ROTableor to set its metatable will throw an error.
- NodeMCU also introduces a concept known as Lua Flash Store (LFS). This enables Lua code (and any string constants used in this code) to be compiled and stored in code space, and hence without using RAM resources. Such LFS functions are still represented by the type function and can be executed just like any other Lua function.
Environments and the Global Environment¶
These are implemented as per the LRM. Note that the Lua 5.1 and Lua 5.3 language implementations are different and can introduce breaking incompatibilities when moving between versions, but this is a Lua issue rather than a NodeMCU one.
All LFS functions, any string constants used in these functions, and any ROTables are stored in static code space. These are ignored by the Lua Garbage collector (LGC). The LGC only scans RAM-based resources and recovers unused ones. The NodeMCU LGC has slightly modified configuration settings that increase its aggressiveness as heap usage approaches RAM capacity.
The firmware includes the full coroutine implementation, but note that there are some slight differences between the standard Lua 5.1 and Lua 5.3 C API implementations. (See Feature breaks below.)
The NodeNCU runtime offers a full implementation of the Lua language as defined in LRM §3 and its subsections.
The Application Program Interface¶
LRM §4 describes the C API for Lua. This is used by NodeMCU modules written in the C source language to interface to the the Lua runtime. The header file
lua.h is used to define all API functions, together with their related types and constants. This section 4 forms the primary reference, though the NodeMCU makes minor changes as detailed below to support LFS and ROTable resources.
Error Handling in C¶
LRM §4.6 describes how errors are handled within the runtime.
The normal practice within the runtime and C modules is to throw any detected errors -- that is to unroll the call stack until the error is acquired by a routine that has declared an error handler. Such an environment can be established by lua_pcall and related API functions within C and by the Lua function pcall; this is known as a protected environment. Errors which occur outside any protected environment are not caught by the Lua application and by default trigger a "panic". By default NodeMCU captures the error traceback and posts a new SDK task to print the error before restarting the processor.
The NodeMCU runtime implements a non-blocking threaded model that is similar to that of
node.js, and hence most Lua execution is initiated from event-triggered callback (CB) routines within C library modules. NodeMCU enables full recovery of error diagnostics from otherwise unprotected Lua execution by adding an additional auxiliary library function
luaL_pcallx. All event-driven Lua callbacks within our library modules use
luaL_pcallx instead of
lua_call. This has the same behaviour if no uncaught error occurs. However, in the case of an error that would have previously resulted in a panic, a new SDK task is posted with the error traceback as an upvalue to process this error information.
The default action is to print a full traceback and then trigger processor restart, that is a similar outcome as before but with recovery of the full error traceback. However the
node.onerror() library function is available to override this default action; for example developers might wish to print the error without restarting the processor, so that the circumstances which triggered the error can be investigated.
Additional API Functions and Types¶
These are available for developers coding new library modules in C. Note that if you compare the standard and NodeMCU versions of
lua.h you will find a small number of entries not listed below. This is because the Lua 5.1 and Lua 5.3 variants are incompatible owing to architectural differences. However,
laubxlib.h includes equivalent wrapper version-compatible functions that may be used safely for both versions.
ROTable and ROTable_entry¶
Extra structure types used in
LROT macros to declare static RO tables. See detailed section below.
void (lua_createrotable) (lua_State *L, ROTable *t, const ROTable_entry *e, ROTable *mt); [-0, +1, -]
Create a RAM based
ROTable pointing to the
e, and metatable
lua_debugbreak and ASSERT¶
ASSERT(condition) are available for development debugging. If
DEVELOPMENT_USE_GDB is defined then these will respectively trigger a debugger break and evaluate a conditional assert prologue on the same. If not, then these are effectively ignored and generate no executable code.
int lua_dump (lua_State *L, lua_Writer writer, void *data, int strip); [-0, +0, –]
Dumps function at the top of the stack function as a binary chunk as per LRM. However the last argument
strip is now an integer is in the range -1..2, rather a boolean as per standard Lua:
- -1, use the current default strip level (which can be set by
- 0, keep all debug info
- 1, discard Local and Upvalue debug info; keep line number info
- 2, discard Local, Upvalue and line number debug info
The internal NodeMCU
Proto encoding of debug line number information is typically 15× more compact than in standard Lua; the intermediate
strip=1 argument allows the removal of most of the debug information whilst retaining the ability to produce a proper line number traceback on error.
int lua_freeheap (void); [-0, +0, –]
returns the amount of free heap available to the Lua memory allocator.
int lua_gc (lua_State *L, int what, int data); [-0, +0, m]
provides an option for
LUA_GCSETMEMLIMIT sets the available heap threshold (in bytes) at which aggressive sweeping starts.
void lua_getlfsconfig (lua_State *L, int *conf); [-0, +0, -]
conf is not
NULL, then this returns an int summary of the LFS configuration. The first 3 items are the mapped and flash address of the LFS region, and its allocated size. If the LFS is loaded then the 4th is the current size used and the 5th (for Lua 5.3 only) the date-timestamp of loaded LFS image.
lua_State * lua_getstate(); [-0, +0, -]
returns the main thread
lua_State record. Used in CBs to initialise
L for subsequent API call use.
void lua_pushrotable (lua_State *L, ROTable *p); [-0, +1, -]
Pushes a ROTable onto the stack.
int lua_stripdebug (lua_State *L, int level); [-1, +0, e]
This function has two modes. A value is popped off the stack.
- If this value is
nil, then the default strip level is set to given
level if this is in the range 0 to 2. Returns the current default level.
- If this value is a Lua function (in RAM rather than in LFS), then the prototype hierarchy within the function is stripped of debug information to the specified level. Returns an approximate estimate of the heap freed (this can be a lot higher if some strings can be garbage collected).
void lua_writestring(const char *s, size_t l); /* macro */
Writes a string
s of length
stdout. Note that any output redirection will be applied to the string.
void lua_writestringerror(const char *s, void *p). /* macro */
Writes an error with CString format specifier
s and parameter
stderr. Note on the ESP devices this error will always be sent to
UART0; output redirection will not be applied.
The Debug Interface¶
int lua_pushstringsarray (lua_State *L, int opt); [-0, +1, m]
Pushes an array onto the stack containing all strings in the specified strings table. If
opt = 0 then the RAM table is used, else if
opt = 1 and LFS is loaded then the LFS table is used, else
nil is pushed onto the stack.
Returns a status boolean; true if a table has been pushed.
Auxiliary Library Functions and Types¶
Note that the LRM defines an auxiliary library which contains a set of functions that assist in coding convenience and economy. These are strictly built on top of the basic API, and are defined in
lauxlib.h. By convention all auxiliary functions have the prefix
luaL_ so module code should only contain
luaL_XXXX() data declarations and functions. And since the
lauxlib.h itself incudes
lua.h, all C modules should only need to
#include "lauxlib.h" in their include preambles.
NodeMCU adds some extra auxiliary functions above those defined in the LRM.
int lua_lfsreload (lua_State *L); [-1, +1, -]
This function pops the LFS image name from the stack, and if it exists and contains the correct image header then it reloads LFS with the specified image file, and immediately restarts with the new LFS image loaded, so control it not returned to the calling function. If the image is missing or the header is invalid then an error message is pushed onto the stack and control is returned. Note that if the image has a valid header but its contents are invalid then the result is undetermined.
int luaL_pcallx (lua_State *L, int narg, int nresults); [-(nargs + 1), +(nresults|1), –]
Calls a function in protected mode and providing a full traceback on error.
nresults have the same meaning as in lua_call. If there are no errors during the call, then
luaL_pcallx behaves exactly like
lua_call. However, if there is any error,
lua_pcallx has already established an traceback error handler for the call that catches the error. It cleans up the stack and returns the negative error code.
Any caught error is posted to a separate NodeMCU task which which calls the error reporter as defined in the registry entry
onerror with the traceback text as its argument. The default action is to print the error and then set a 1 sec one-shot timer to restart the CPU. (One second is enough time to allow the error to be sent over the network if redirection to a telnet session is in place.) If the
onerror entry is set to
Note that the Lua runtime does not call the error handler if the error is an out-of memory one, so in this case the out-of-memory error is posted to the error reporter without a traceback.
int luaL_posttask (lua_State* L, int prio); [-1, +0, e]
Posts a task to execute the function popped from the stack at the specified user task priority
prio one of:
Note that the function is invoked with the priority as its parameter.
int luaL_pushlfsmodule ((lua_State *L); [-1, +1, -]
This function pops a module name from the stack. If this is a string, LFS is loaded and it contains the named module then its closure is pushed onto the stack as a function value, otherwise
nil is pushed.
Returns the type of the pushed value.
int luaL_pushlfsmodules (lua_State *L); [-0, +1, m]
If LFS is loaded then an array of the names of all of the modules in LFS is pushed onto the stack as a function value, otherwise
nil is pushed.
Returns the type of the pushed value.
int luaL_pushlfsdts (lua_State *L); [-0, +1, m-]
If LFS is loaded then the Unix-style date-timestamp for the compile time of the image is pushed onto the stack as a integer value, otherwise
nil is pushed. Note that the primary use of this stamp is to act as a unique identifier for the image version.
Returns the type of the pushed value.
void (luaL_reref) (lua_State *L, int t, int *ref); [-1, +0, m]
Variant of luaL_ref. If
*ref is a valid reference in the table at index t, then this is replaced by the object at the top of the stack (and pops the object), otherwise it creates and returns a new reference using the
int (luaL_rometatable) (lua_State *L, const char* tname, const ROTable *p); [-0, +1, e]
luaL_newmetatable() for ROTable metatables. Adds key / ROTable entry to the registry
[tname] = p, rather than using a new RAM table.
This macro executes
luaL_unref(L, t, r) and then assigns
r = LUA_NOREF.
Declaring modules and ROTables in NodeMCU¶
All NodeMCU C library modules should include the standard header "
module.h". This internally includes
lnodemcu.h and these together provide the macros to enable declaration of NodeMCU modules and ROTables within them. All ROtable support macros are either prefixed by
LRO_ (Lua Read Only) or in the case of table entries
NODEMCU_MODULE(sectionname, libraryname, map, initfunc)
This macro enables the module to be statically declared and linked in the
ROM ROTable. The global environment's metafield
__index is set to the ROTable
ROM hence any entries in the ROM table are resolved as read-only entries in the global environment.
sectionname. This is the linker section for the module and by convention this is the uppercased library name (e.g.
FILE). Behind the scenes
_module_selected is appended to this section name if corresponding "use module" macro (e.g.
LUA_USE_MODULES_FILE) is defined in the configuration. Only the modules sections
*_module_selected are linked into the firmware image and those not selected are ignored.
libraryname. This is the name of the module (e.g.
file) and is the key for the entry in the
map. This is the ROTable defining the functions and constants for the module, and this is the corresponding value for the entry in the
initfunc. If this is not NULL, it should be a valid C function and is call during Lua initialisation to carry out one-time initialisation of the module.
LROT_BEGIN and LROT_END¶
These macros start and end a ROTable definition. The three parameters must be the same in both declarations.
rt. ROTable name.
mt. ROTable's metatable. This should be of the form
LROT_TABLEREF(tablename) if the metatable is used and
flags. The Lua VM table access routines use a
flags field to short-circuit where the access needs to honour metamethods during access. In the case of a static ROTable this flag bit mask must be declared statically during compile rather than cached dynamically at runtime. Hence if the table is a metatable and it includes the metamethods
__eq, then the mask field should or-in the corresponding mask for example if
__index is used then the flags should include
LROT_MASK_INDEX. Note that index and GC are a very common combination, so
LROT_MASK_GC_INDEX is also defined to be
(LROT_MASK_GC | LROT_MASK_INDEX).
ROTables only support static declaration of string keys and value types: C function, Lightweight userdata, Numeric, ROTable. These are entries are declared by means of the
LROT_TABENTRY macros. All take two parameters: the name of the key and the value. For Lua 5.1 builds
LROT_INTENTRY both generate a numeric
TValue, but in the case of Lua 5.3 these are separate numeric subtypes so these macros generate the appropriate subtype.
Note that ROTable entries can be declared in any order except that keys starting with "
_" must be declared at the head of the list. This is a pragmatic constraint for runtime efficiency. A lookaside cache is used to optimise key searches and results in a direct table probe in over 95% of ROTable accesses. A table miss (that is the key doesn't exist) still requires a full scan of the list, and the main source of table misses are scans for metafield values. Forcing these to be at the head of the ROTable allows the scan to abort on reading the first non-"
ROTables can still support other key and value types by using an index metamethod to point at an C index access function. For example this technique is used in the
utf8 library to return
LROT_BEGIN(utf8_meta, NULL, LROT_MASK_INDEX) LROT_FUNCENTRY( __index, utf8_lookup ) LROT_END(utf8_meta, NULL, LROT_MASK_INDEX) LROT_BEGIN(utf8, LROT_TABLEREF(utf8_meta, 0) LROT_FUNCENTRY( offset, byteoffset ) LROT_FUNCENTRY( codepoint, codepoint ) LROT_FUNCENTRY( char, utfchar ) LROT_FUNCENTRY( len, utflen ) LROT_FUNCENTRY( codes, iter_codes ) LROT_END(utf8, LROT_TABLEREF(utf8_meta), 0)
Any reference to
utf8.charpattern will call the
__index method function (
utf8_lookup()); this returns the UTF8 character pattern if the name equals
nil otherwise. Hence
utf8 works as standard even though ROTables don't natively support a string value type.
- Basic Lua functions, coroutine support, Lua module support, string and table manipulation are as per the standard Lua implementation. However, note that there are some breaking changes in the standard Lua string implementation as discussed in the LRM, e.g. the
\zend-of-line separator; no string functions exhibit a CString behaviour (that is treat
"\0"as a special character).
- The modulus operator is implemented for string data types so
str % varis a synonym for
str % tblis a synonym for
string.format(str, table.unpack(tbl)). This python-like formatting functionality is a very common extension to the string library, but is awkward to implement with
stripparameter can take integer values 1,2,3 (the
lua_stripdebugstrip parameter + 1).
falseis synonymous to
3and omitted takes the default strip level.
stringlibrary does not offer locale support.
- The 5.3
mathlibrary is expanded compared to the 5.1 one, and specifically:
- Not implemented:
- Input, output OS Facilities (the
oslibraries) are not implement for firmware builds because of the minimal OS supported offered by the embedded run-time. The separately documented
nodelibraries provide functionally similar analogues. The host execution environment implemented by
luac.crossdoes support the
- The full
debuglibrary is implemented less the
- An extra function
debug.getstrings(type)has been added;
typeis one of
'RAM'(the default). Returns a sorted array of the strings returned from the
- An extra function
Standard Lua has a number of breaking incompatibilities that require conditional code to enable modules using these features to be compiled against both Lua 5.1 and Lua 5.3. See Lua 5.2 §8 and Lua 5.3 §8 for incompatibilities with Lua 5.1.
A key strategy in our NodeMCU migration to Lua 5.3 is that all NodeMCU application modules must be compilable and work under both Lua versions. This has been achieved by three mechanisms
- The standard Lua build has conditionals to enable improved compatibility with earlier versions. In general the Lua 5.1 compatibilities have been enabled in the Lua 5.3 builds.
- Regressing new Lua 5.3 features into the Lua 5.1 API.
- For a limited number of features the Project accepts that the two versions APIs are incompatible and hence modules should either avoid their use or use
#if LUA_VERSION_NUM == 501 conditional compilation.
The following subsections detail how NodeMCU Lua versions deviate from standard Lua in order to achieve these objectives.
Enabling Compatibility modes¶
The following Compatibility modes are enabled:
ìntcast are still used within NodeMCU modules.
LUA_COMPAT_UNPACK. This retains
ROM.unpack as a global synonym for
LUA_COMPAT_LOADERS. This keeps
package.loaders as a synonym for
LUA_COMPAT_LOADSTRING. This keeps
loadstring(s) as a synonym for
New Lua 5.3 features back-ported into the Lua 5.1 API¶
- Table access routines in Lua 5.3 are now type
voidand return the type of the value pushed onto the stack. The 5.1 routines
luaL_getmetatablehave been updated to mirror this behaviour and return the type of the value pushed onto the stack.
- There is a general numeric comparison API function
lua_compare()with macros for
lua_lessthan()whereas 5.1 only support the
<tests through separate API calls. 5.1 has been update to mirror the 5.3. implementation.
- Lua 5.3 includes a
lua_absindex(L, idx)which converts ToS relative (e.g.
-1) indices to stack base relative and hence independent of further push/pop operations. This makes using down-stack indexes a lot simpler. 5.1 has been updated to mirror this 5.3 function.
\0is now a valid pattern character in search patterns, and the
%zpattern is no longer supported. We suggest that modules either limit searching to non-null strings or accept that the source will require version variants.
The stack pseudo-index
LUA_GLOBALSINDEXhas been removed. Modules must either get the global environment from the registry entry
LUA_RIDX_GLOBALSor use the
lua_getglobalAPI call. All current global references in the NodeMCU library modules now use
Shared userdata and table upvalues. Lua 5.3 now supports the sharing of GCObjects such as userdata and tables as upvalue between C functions using
lua_setupvalue. This has changed from 5.1 and we suggest that modules either avoid using these API calls or accept that the source will require version variants.
The environment support has changed from Lua 5.1 to 5.3. We suggest that modules either avoid using these API calls or accept that the source will require version variants.
The coroutine yield / resume API support has changed in Lua 5.3 to support yield and resume across C function calls. We suggest that modules either avoid using these API calls or accept that the source will require version variants.