Your Input Matters: How Do You Use STM32Cube Examples?

[Emil Damkjaer PETERSEN]

Dear STM32 Community,

 

We believe that the example projects provided for the STM32Cube embedded software are a valuable resource when developing projects.

 

We would like to better understand how developers use these examples in their workflow and explore ways to improve them. Therefore, we have a few questions we would like to ask:

 

From where do you obtain the examples, and why?

• Are you working with the examples through the STM32Cube MCU packages, the STM32CubeMX Example Selector, or through GitHub?
• Is there a particular reason you prefer one channel over the others?

How do you use the examples in your work with STM32 embedded software?

• Do you use the examples to learn how to use a driver or a feature, as reference code for implementation, and/or for debugging?

What is working well, and what can we do to improve the examples and/or your experience working with them?

 

You can either reply directly in this thread or feel free to send me a private message.

 

Best regards,
Emil

[Radosław]

More content without HAL please ;)

[Emil Damkjaer PETERSEN]

Hello @Radosław 

Are you missing LL examples? If yes, what types of examples are you missing?

 

Best Regards,

Emil

[Radosław]

Emil thank You for response.

 

LL Is still HAL, all register names are changed, and each CPU package have only few of them.

I'm was very happy to see snippets for STM32L0, as a packege and at the and of Reference Manual.

For Cube Hal St provides many Aplications note, but i'm still waiting for thinks like cookbooks (AN4539 for example).

Hal gives easy and quick start, but any knowledge about MCU architecture, posible option even not standard or typical,

 

Additionally, for example,  In hal ST provides functrions for dlash programing that need to be called from RAM,

 

ex. __RAM_FUNC HAL_StatusTypeDef HAL_FLASHEx_ProgramParallelHalfPage(uin .....

 

There is no explanation WHY, additionally Linker for GCC do not have proper section for that, (so example is incomplete).

 

 

 

 

[Harvey White]

I find that I cannot use most of the examples for several reasons.

1. You use BSP in a lot of examples.  I don't.  What I have to do is to work around your examples.  Since I don't use BSP, integrating your example code into my projects to make use of it is a lot of work.  
2. If you use LL drivers, I don't.  I use HAL drivers, which complicates things a bit.
3. I use FreeRTOS, and have a well developed ecostructure.  Your examples need to be adapted.
4. As is normal, the examples don't do what I need, so I need to deduce what I can.
5. Many of the examples are more complex than needed.  (As an example, look at Borland Delphi examples which when reading an array, demonstrates 12 other things and uses 15 other features.  This does not help).

Now, the github examples I don't find useful at all, they generally don't solve what I need to solve.

The example projects, though, I find somewhat frustrating.  You create example projects for ST boards, not at all unreasonable, but neglect (likely because you use BSP drivers) a bare bones, "It's not our hardware configuration" custom boards.

Sometimes, your examples assume too much, because you, of course, know exactly what you're doing.  As I've mentioned in the past, every program is written with an assumption of "of course they'll write it like that to do it this way."  When the assumptions you make are not the same as the person reading the example, then there's a problem.

I never, for example, got Touch-GFX to work well, for both that reason and the fact that the two tool chains were not well integrated when I tried them.

Hope this helps a bit.

 

[Emil Damkjaer PETERSEN]

Hello @Harvey White ,

 

Thanks a lot for your thorough answer, it helps a lot :) 

For some of the points you made, i have a few follow up questions:

---

I use FreeRTOS, and have a well developed ecostructure.  Your examples need to be adapted

Is this due to missing examples around FreeRTOS, or general you not being able to reuse or examples, since they require to much work to fit in your own ecosystem?

 

---

As is normal, the examples don't do what I need, so I need to deduce what I can.

Would you be able to give more info? Would like to understand why this is the case, and how we can do it different to help?

 

Finally it would also be very appreciated if you can describe what your challenges with GitHub examples are.

 

Again thanks a lot for your input.

Best Regards,

Emil

 

[Harvey White]

Thanks.  Let's see if I can address the points:

--------------------------------------------------------

For some of the points you made, i have a few follow up questions:

I use FreeRTOS, and have a well developed ecostructure.  Your examples need to be adapted

Is this due to missing examples around FreeRTOS, or general you not being able to reuse or examples, since they require to much work to fit in your own ecosystem?

---------------------------------------------------------

Let's take the HAL_SPI drivers:  They work well enough, although some have complained about excess code.  

In theory, they have the equivalent of semaphores to keep the code well behaved in a multi threaded operating system, they're not explained.  (I regard HAL code to be poorly commented, but that's just personal)  In actuality, I have to put semaphores around them to keep everything well behaved.  In another specific example, many displays (ILI9341 chip, 320 * 240 color TFT) cannot be used without some modification to your drivers.  Your drivers assume that (hardware wise) you assert CS, transmit or receive, release CS.  The ILI9341 needs CS asserted, command bytes sent, data bytes sent, and then CS released.  Can't use the hardware to do this, need to program it.  I also am programming in C++, since the graphics system I wrote won't work otherwise.

int32_t HAL_SPI::Send(
						uint8_t* cmdptr,
						uint32_t cmd_count,
						uint8_t* dataptr,
						uint32_t data_count,
						bool USE_CS,
						enum SPI_TRANSFER_MODE Override)
{

	enum SPI_TRANSFER_MODE		NEW_mode = OUT_mode;
//	int							old_divisor;

	// ********************************** check input parameters***********************************
	// need to check overrides
	switch (Override)
	{
		case SPI_DMA:			// TRANSMIT IS DMA FROM BUFFER, RECEIVE IS IRQ QUEUE (MAY CHANGE)
		case SPI_IRQ:			// TRANSMIT IS IRQ DRIVEN
		case SPI_BLOCKING:		// TRANSMIT AND RECEIVE ARE BYTE BLOCKING
		{
			NEW_mode = Override;
			break;
		}
		default:
		{
			NEW_mode = OUT_mode;
			// leave mode as is, cannot override
		}
	}

	switch (NEW_mode)
	{
		case SPI_BLOCKING:		// TRANSMIT AND RECEIVE ARE BYTE BLOCKING
		{
			time_start = _MICROSECOND_TIMER.Instance->CNT;
			// always semaphore protected
			if (!dedicated) xSemaphoreTake(busy, portMAX_DELAY);
			// direct call to STMICRO driver, blocking mode
			hspi->Instance->CR1 &= ~SPI_BAUDRATEPRESCALER_256;// reset to highest baudrate
			hspi->Instance->CR1 |= transmit_divisor;

			if (RESET.port != nullptr) RESET.set();				// make sure reset if high
			if (USE_CS) RESET_CS();								// CS low if needed
			if (cmdptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.reset();
				result = HAL_SPI_Transmit(hspi, cmdptr, cmd_count, timeout);
				// place to add delay if available
			}
			if (dataptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.set();
				result = HAL_SPI_Transmit(hspi, dataptr, data_count, timeout);
				// place to add delay if available
			}
			// error reporting
			if (USE_CS) SET_CS();								// CS high if needed
			ERROR_CODE = hspi->ErrorCode;
			ERROR_REASON = (enum COMM_ERROR_TYPE) ERROR_CODE;
			operations++;
			// record errors if needed
			switch (result)
			{
				case HAL_ERROR: 					//= 0x01U,
				{
					fail++;
					break;
				}
				case HAL_BUSY: 					//= 0x02U,
				{
					fail++;
					break;
				}
				case HAL_TIMEOUT: 					//= 0x03U
				{
					fail++;
					break;
				}
				case HAL_OK:
				{
					// was OK
					succeed++;
					break;
				}
			}

			// return semaphore for next operation
			if (!dedicated) xSemaphoreGive(busy);
			time_stop = _MICROSECOND_TIMER.Instance->CNT;
			delta_time = time_stop - time_start;
			break;
		}
		case SPI_QUEUE:			// TRANSMIT QUEUE SENDS
		{
			// always semaphore protected
			if (!dedicated) xSemaphoreTake(busy, portMAX_DELAY);
			for (uint32_t i = 0; i < data_count; i++)
			{
				// send data
				xQueueSend(send_queue, &dataptr, portMAX_DELAY);
				dataptr++;
			}
			operations++;
			// return semaphore for next operation
			if (!dedicated) xSemaphoreGive(busy);
			break;
		}
		case SPI_IRQ:			//
		{
			time_start = _MICROSECOND_TIMER.Instance->CNT;
			// always semaphore protected
			if (!dedicated) xSemaphoreTake(busy, portMAX_DELAY);
			// direct call to STMICRO driver, blocking mode
			xSemaphoreTake(done, portMAX_DELAY);				// if can take, no active DMA

			hspi->Instance->CR1 &= ~SPI_BAUDRATEPRESCALER_256;// reset to highest baudrate
			hspi->Instance->CR1 |= transmit_divisor;

			if (RESET.port != nullptr) RESET.set();				// make sure reset if high
			if (USE_CS) RESET_CS();								// CS low if needed
			if (cmdptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.reset();
				result = HAL_SPI_Transmit(hspi, cmdptr, cmd_count,timeout);
				// place to add delay if available
			}
			if (dataptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.set();
				result = HAL_SPI_Transmit_IT(hspi, dataptr, data_count);
				xSemaphoreTake(done, portMAX_DELAY);
			}
			// error reporting
			if (USE_CS) SET_CS();								// CS high if needed
			ERROR_CODE = hspi->ErrorCode;
			ERROR_REASON = (enum COMM_ERROR_TYPE) ERROR_CODE;
			operations++;
			// record errors if needed
			switch (result)
			{
				case HAL_ERROR: 					//= 0x01U,
				{
					fail++;
					break;
				}
				case HAL_BUSY: 					//= 0x02U,
				{
					fail++;
					break;
				}
				case HAL_TIMEOUT: 					//= 0x03U
				{
					fail++;
					break;
				}
				case HAL_OK:
				{
					// was OK
					succeed++;
					break;
				}
			}

			// return semaphore for next operation
			xSemaphoreGive(done);
			if (!dedicated) xSemaphoreGive(busy);
			time_stop = _MICROSECOND_TIMER.Instance->CNT;
			delta_time = time_stop - time_start;
			break;
		}
		case SPI_DMA:			// TRANSMIT IS DMA FROM BUFFER, RECEIVE IS IRQ QUEUE (MAY CHANGE)
		{
			time_start = _MICROSECOND_TIMER.Instance->CNT;
			// always semaphore protected
			if (!dedicated) xSemaphoreTake(busy, portMAX_DELAY);
			// direct call to STMICRO driver, blocking mode
			xSemaphoreTake(done, portMAX_DELAY);				// if can take, no active DMA

			hspi->Instance->CR1 &= ~SPI_BAUDRATEPRESCALER_256;// reset to highest baudrate
			hspi->Instance->CR1 |= transmit_divisor;

			if (RESET.port != nullptr) RESET.set();				// make sure reset if high
			if (USE_CS) RESET_CS();								// CS low if needed
			if (cmdptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.reset();
				result = HAL_SPI_Transmit(hspi, cmdptr, cmd_count,timeout);
//				xSemaphoreTake(done, portMAX_DELAY);
				// place to add delay if available
			}
			if (dataptr != nullptr)
			{
				// command pointer says to send data with A0 down
				if (A0.port != nullptr) A0.set();
				result = HAL_SPI_Transmit_DMA(hspi, dataptr, data_count);
				xSemaphoreTake(done, portMAX_DELAY);
			}
			// error reporting
			if (USE_CS) SET_CS();								// CS high if needed
			ERROR_CODE = hspi->ErrorCode;
			ERROR_REASON = (enum COMM_ERROR_TYPE) ERROR_CODE;
			operations++;
			// record errors if needed
			switch (result)
			{
				case HAL_ERROR: 					//= 0x01U,
				{
					fail++;
					break;
				}
				case HAL_BUSY: 					//= 0x02U,
				{
					fail++;
					break;
				}
				case HAL_TIMEOUT: 					//= 0x03U
				{
					fail++;
					break;
				}
				case HAL_OK:
				{
					// was OK
					succeed++;
					break;
				}
			}

			// return semaphore for next operation
			xSemaphoreGive(done);
			if (!dedicated) xSemaphoreGive(busy);
			time_stop = _MICROSECOND_TIMER.Instance->CNT;
			delta_time = time_stop - time_start;
		break;
		}
		default:
		{
			// do nothing
		}
	} 							// end out mode switch
	return HAL_OK;
}

The driver is designed to adapt to several situations:

HAL mode can be either blocking, IRQ, or DMA depending on setup.

SPI clock speed is set for each device.

Driver can set a command field and a data field.

A0 line is used to differentiate between them, which is defined (from the .IOC file) in setup.  it can be omitted and either command or data fields can be omitted, defaulting to a single transmit sequence.  Commands are short and sent blocking.  Data is longer and may be sent with DMA.

Semaphores protect the code operation unless the interface is dedicated, in which case, at this level, busy semaphores are not checked.  There's a busy at a higher level, so the overall driver is still protected.  Busy at this level allows the interface to be shared (ex: Touchscreen and SD card).

The interface can be set up to run in an override mode, which allows short commands to be blocking I/O, and data transfers (writing to the display) to be DMA.  Set up the driver for DMA and then override with blocking mode.

The particular answer to your question is the use of semaphores.  None of your code allows for that.  Granted, you're pushing AZURE, but AZURE has a very awkward queue mechanism (to me) and performs badly with C++ code, in my experience.

I've managed to work around the quirks in FreeRTOS, and even managed to piece together how to get Tracealyzer to work.  

I could suggest that in the HAL code, you might want to put in a comment such as:

// SYSTEM SEMAPHORE may be needed for threadsafe operation
// insert take semaphore operation here to restrict access
// end SYSTEM SEMAPHORE take

................... HAL code ........................

// if using SYSTEM SEMAPHORE
// add code to release semaphore
// end SYSTEM SEMAPHORE give

 

As is normal, the examples don't do what I need, so I need to deduce what I can.

Would you be able to give more info? Would like to understand why this is the case, and how we can do it different to help?

As in the above example, the hardware CSS signal cannot be controlled without rewriting your chip support routine, so it needed a wrapper and I had to forgo using the hardware CSS signal.  I had to figure out how your CSS signal was generated and where in the code.  I don't really expect your HAL drivers to give this level of flexibility, but it's what I need.  I don't trust the use of the lock you put in the HAL driver, not sure that it works, and I've had instances where it didn't.  Your code is not written for multiple threads.

and lastly:

Finally it would also be very appreciated if you can describe what your challenges with GitHub examples are

For one, the GitHub examples may use LL drivers, which is fine, but, as above, are not written with the idea of an operating system.  My code is generally parametrically driven, and I had enough fun writing graphics drivers at the chip level.  I'd rather the support code take care of that, but in an understandable manner (comments!).

Hope this answers the questions.

[ERROR]

Hi @Emil Damkjaer PETERSEN 

> From where do you obtain the examples, and why?
All of them depending on purposes. STM32Cube based projects use STM32Cube MCU packages. If I need to adapt a project for another MCU or board, I prefer to fork your github project and keep my changes under version control.

 

>How do you use the examples in your work with STM32 embedded software?

All of that your have listed.

 

>What is working well, and what can we do to improve the examples and/or your experience working with them?

Your support works well here at community.st.com.

Regarding improvements.

Some projects have no ioc-files. It makes difficult using them as reference code.

It would be nice if you contribute to the TinyUSB project to support your chips with two USBs (STM32N6 etc).

 

[MA4]

Hello,

Yes, it's really useful.
I use Github, thanks to it I can understand how some peripherals works and I compare my code with ST's and see what I'm doing wrong. I use Git to not have 2 IDE and avoid to use computer storage.

I don't use to much examples in IDE because sometimes is not quick to adapt for my electronics card. Because of external input/output and wrong CPU for example. But we use it to test new components thanks to demo board too.

Keep doing many different examples please :D

[chornbeck]

The example code provided by y'all is very helpful, but almost entirely to be torn down. I agree with the other commentor that you rely too heavily on the BSP code for certain boards. For example, when developing camera applications for the STM32N6 recently, I had to dig heavily into the example code, because it relied so heavily on the BSP code for the Discovery Kit board. I was using a custom-spun board with the N6 as the centerpiece, but all of the examples assumed very specific setups about the DK, or used obfuscated BSP calls that jumped over a ton of steps, skipped over lots of HAL calls, etc.

I have also encountered, multiple times, instances where the example code contains a Cube MX bug fix or workaround that isn't mentioned anywhere, so when I try to recreate the project bones in Cube MX, it doesn't work due to these little tricks that are probably obvious for ST employees working with the chip for months.

I am also not a fan of the examples being primarily ST Cube IDE based. The way the file structures are set up in that IDE are very different than anything else, and it takes a significant amount of work, to the point where I'd rather recreate the project elsewhere, to attempt to migrate it to a CMake project for general development. I understand the "Oh need to use our own IDE" but there has to be a limit where it's understood that most developers use a different environment like VSCode, and a different build system.

[Harvey White]

I'll agree.  Perhaps a different example code schema would be to write your code, then in a specific section do something like:

//*********************************************************************************
// Code here to write one character to the USART, assuming that the USART is already initialized
// example for HAL and CUBEMX using blocking send:
result = HAL_UART_Transmit(huart, dataptr, data_count, 20);
if (LF) result = HAL_UART_Transmit(huart, (uint8_t*)&terminate[0], sizeof(terminate), 20);
// ********************************************************************************

Using a usart send as an example.

Your BSP packages are so specific to a board as to limit them only to running a demo on your specific hardware.

You have experts in BSP packages, experts in LL packages, and experts in HAL packages.  Simple #ifdef statements can control which gets used, and each expert can contribute their own connection to the driver.

 

 

[Hamady]

Hello,

As other mentioned BSP package can be a nightmare.

When you devlopp a custom PCB you can spend a lot of time debugging code !!!

Other when using DMA2D on STM32N6 after one week of debugging i had the not pleasant surpise to discover that the example had code put in the USER CODE section to make it work .

I think that examples should mention when code for driver other than cube MX is used !!!!!

[Emil Damkjaer PETERSEN]

Hello @Harvey White, @chornbeck, @Hamady  and @john37 

 

How would I fit with your way of work, if we changed the BSP code with code generated from STM32CubeMX and application code with a clear separation between the two, by having MX generated code in one set of files that the application code calls, in a structure like this?:

Best Regards,

Emil

[Harvey White]

I'd be concerned that as long as I have to go through and rewrite all the low level calls, that I need to do this only one time and let the driver (or a global) figure out which driver to call.

From what I can see, the only use for a BSP package is to run pre packaged demos that do not (IMHO) contribute to learning much, if anything.  The problem is that you do not generate BSP code anywhere.  I have to dig the code out of the examples, and IIRC, the examples are not well documented, and are not written to play nice with other code.  The HAL2 drivers at least allow you to make a choice between the LL and the HAL level drivers.

Given that choice. I really don't care whether or not when I call a HAL2 driver if it uses LL drivers or higher level HAL2 drivers, as long as it works.  If you shifted to BSP drivers, I wouldn't notice as long as they were more or less compatible with the HAL drivers (in terms of function).

I tend to be far more interested in the "get it done" approach to drivers, as long as they do what I need to do.  I generally don't care what's under the hood.  

My software uses a lot of #ifdef to control operation. Want a serial port, change the #ifdef (uncomment it), and so on.  You may consider the same approach.  

What I'd like to see, given that you support more than one OS, is a definition that switches that OS in and out as needed.  

I do like the idea of a mode on a driver to use DMA or not, IRQ or not, blocking or not, and if an OS is defined, a queue or not.  These choices are not useful for all driver situations (SPI, I2C, Serial are all I use), but a good HAL2 driver should address these (and quite frankly, if you use the IOC file, then please put in the callbacks as needed).

 

[Emil Damkjaer PETERSEN]

@john37, for references could it be referring some of our own documentation?

We are putting in place new type of online doc, and could therefore make references like this one to our ADC (https://dev.st.com/stm32cube-docs/stm32u5-hal2/2.0.0-beta.1.1/docs/drivers/hal_drivers/adc/hal_adc_overview.html#hal-adc-overview) as an example. Also to the relevant API description e.g. ADC START (https://dev.st.com/stm32cube-docs/stm32u5-hal2/2.0.0-beta.1.1/docs/drivers/hal_drivers/adc/api/hal_adc_exported_functions.html#_CPPv413HAL_ADC_StartP16hal_adc_handle_t)

 

Best Regards,

Emil

[Harvey White]

I liked the idea so much that I decided to see if it would work.  I have a partial implementation using embOS and FreeRTOS.  Full testing of emBOS is going to require a lot of work, but the implementation should be more or less straight forward.  For how to do the actual substitutions, I'm using the instructions from Segger:  https://www.segger.com/products/rtos/embos/ 

File wise, OS_MACROS.h contains the main data, and HAL_I2C shows how to paste in the definitions of the OS handles, and the part in HAL_I2C.cpp shows how to insert the code.  It's still in progress, though.  The normal FreeRTOS views do show the proper creation of the semaphore.

For the macros:

/*******************************************************************************************************************
 *	MACRO SUBSTITUTIONS
 *	INCLUDE FILES
 *
 ******************************************************************************************************************/



// SEMAPHORES
//

// all FreeRTOS macros
#ifdef _FREERTOS

	// semaphore:
	// _WHICH is SemaphoreHandle_t, _DELAY not implemented since this is blocked
	// semaphore is created and given, so that the next _SEMAPHORE_GET will work automatically
	// this is a binary semaphore, non-counting

	#define _SEMAPHORE_CREATE(_WHICH) 	_WHICH = xSemaphoreCreateMutex(); xSemaphoreGive(_WHICH);
	#define _SEMAPHORE_GET (_WHICH, _DELAY) xSemaphoreTake(_WHICH, DELAY);
	#define _SEMAPHORE_RELEASE (_WHICH) xSemaphoreGive(_WHICH);

#endif

which is how the FreeRTOS would work.  Note that this and the embOS implementations make significant compromises because of the different way the two OS treat a semaphore.

#ifdef _EMBOS
	// semaphore:
	// _WHICH is OS_SEMAPHORE*, _DELAY in macro is ignored
	// _WHICH is a pointer to OS_MUTEX

	#define _FOREVER				0
	#define _SEMAPHORE_CREATE(_WHICH) 	 OS_SEMAPHORE_Create(_WHICH); OS_SEMAPHORE_GiveMax(_WHICH,1);			// uses macro for zero count
	#define _SEMAPHORE_GET (_WHICH, _DELAY) OS_SEMAPHORE_TakeBlocked(_WHICH);	// delay is ignored
	#define _SEMAPHORE_RELEASE (_WHICH) OS_SEMAPHORE_Give(_WHICH);

#endif

 and to illustrate the creation of the semaphore:

	_SEMAPHORE_CREATE(busy);


//#ifdef _FREERTOS_STATIC
//	busy = xSemaphoreCreateBinaryStatic( &static_map.I2C_STATIC[(uint8_t)which].BSemaphoreBuffer );
//#else
//	busy = xSemaphoreCreateBinary();
//#endif
//

and for the .hpp file, which is made from a template.

		// *********************************************** SEMAPHORE ******************************************************
#ifdef _FREERTOS
	SemaphoreHandle_t							busy;
#endif

#ifdef _EMBOS
	OS_SEMAPHORE								busy;
#endif

Design notes:  

Names of the macros are chosen to  (hopefully) conflict with no existing names.  Note that enabling _EMBOS and disabling _FREERTOS should take care of everything.

Some of the arguments are ignored, which might give flexibility from OS to OS.  See the notes in the OS_MACRO.h file for the reasoning

Looks as if it might work well, as long as the differences between OS can be ironed out.

 

[Harvey White]

Concept seems to work just fine.

However..........

embOS and FreeRTOS are apparently determined to be very different from each other, number of arguments, and what is returned.

Time for Plan B, 

Replace the macros by functional programming.

More to come.

 

[skidadddytn]

Primarily, I usually like to get the examples from the latest MCU packages so I can also use that folder to "grep" through when I am looking for something.  Never tried the STM32CubeMX Example selector.   I may have accessed them through github a couple of times.  I like the idea of being able to see the rev history for them in github.

I usually run the examples first as-is to learn how the HAL calls work together, then I use it as a reference for writing what I need (which will differ).

I feel like the examples usually work, but If I could pick an area where I would like more examples it would definitely be "low power" (sleep, stop, etc, and multiple modes here).  This area seems to have the most variation when jumping from different stm32 mcus (lots of nuances and differences).