Package management, why to copy all files?

[JKapp.1]

I'm highly confused. Coming from Keil MDK, I really enjoyed the CMSIS-Pack approach to manage dependencies. A MCU (and vendor) independent and efficient approach which works in command line and various IDEs (Keil MDK, VSCode with plugin, Eclipse). See Arm Keil Microcontroller Development Kit (MDK) Getting Started Guide, Arm Keil | CMSIS Packs, CMSIS: Introduction, and Arm Keil | Arm CMSIS.

The key advantage was that there's a global config file to define all dependencies (like CMSIS-Core vxxx, or BSP, etc) in a certain version and then the package manager downloads these packages on demand. There's no need to check in all libraries/dependencies into the source control or add them to the source tree at all - they are just linked automatically.

Now that the ST packages all use CMSIS, why do all examples contain hundreds of MB of libraries that could be installed via a package manager? That feels like a huge step back.

Maybe I'm misunderstanding something, there's the "Embedded Software Packages" menu in CubeIDE, but I can't add packages to the current project, so how are they supposed to be used?

Thanks,

Jan

[Andrew Neil]

An advantage of copying is that your project is self-contained.

It can also mean that you can't get hit by unexpected updates breaking things!

But CubeMX/CubeIDE does give you the option to link rather than copy:

 

[JKapp.1]

Hi @Andrew Neil 

I'm not interested in self-containing folders with 200 MB of libs and 1 MB of source code. I don't want to have that in my source control. I don't want to copy folders and run build on them - I'm assuming that a proper toolchain and package manager are installed (better: are containerized and run in CI/CD), that's the basis of any professional firmware development IMO.

I don't get hit by any unexpected updates when using a package manager with fixed version, that's the whole thing, reproducibility.

Linking is not the right thing, it just refers to a location on the disk, right? Is there an option to define the dependencies along the cproject so that a new user opens that file and just gets asked to download the dependencies? Is it possible to change the version of a lib easily? In Keil this just works out of the box.

Am I getting you right, that there's no option in CubeIDE to add packages? Everything is done in CubeMX? If yes what about this (or other Cube support packages): STMicroelectronics/fp-sns-datalog2: The FP-SNS-DATALOG2 function pack represents an evolution of FP-SNS-DATALOG1 and provides a comprehensive solution for saving data from any combination of sensors and microphones configured up to the maximum sampling rate. Please check st.com where a more recent version of the software might be available.

I can't open that with CubeMX, it's just a CubeIDE project - so it's not possible to change to referenced/linked libraries?

Thanks!

[Andrew Neil]

---

@JKapp.1 wrote:

I'm not interested in self-containing folders 

---

But some people are - which is why it's always a choice!

Personally, it's the way I prefer - so I haven't looked into the alternative under CubeIDE/CubeMX

[JKapp.1]

Does not feel like a real choice - all examples are the "self-containing" option. Sure that's easier to share and especially for beginners that's nice to start but I'm looking for a professional and efficient solution.

My employer and the DevOps team would loudly laugh at me if I told them to either hard link dependencies manually or check them in to git for every single project. And I agree, that's not the right approach (for us).

Three questions:

• Can packages be added to the project using a package manager or not?
• Is CubeMX required for that?
• Any chance to convert just a folder structure like https://github.com/STMicroelectronics/fp-sns-datalog2 to a CubeMX based project to migrate to referenced libs?

Thanks.

[TDK]

What you want is not supported in the current CubeMX framework. You can add links to folders if you want, and put files there, but there is no "package manager" like feature that will automatically link and download the version of the library you want.

The "Embedded Software Packages" get downloaded into a main repository folder and from there are copied to your project folder on code generation. If you want to do this manually, you can link to the repository folder instead, but this would be a manual step.

[Pavel A.]

>  there is no "package manager" like feature that will automatically link and download the version of the library you want.

Not in the ST offering, but users are free to bring their own package/repository management. ST did a great job re-structuring and re-packaging the STM32 libraries on github. Now the CMCIS device part is in a separate git sub-repo (module), "HAL drivers" and middlewares are in separate sub-repos, and more components are available in the "software pack manager".  There are known tools to manage multiple repositories of different nature (git, lfs, artifactory....). Of course it takes time and effort to create a project with that. ST makes chips, it would be naive to expect from them to supply this functionality. On the other hand, Keil, IAR provide software solutions that are worth their price.

So back to STM32: one can pull the STM32 libraries from github as submodules, and make the app project link/refer to these libraries. They won't pollute the user app project. But there is  a prominent warning in the "HAL" library documentation about dependency on certain CMSIS version! You cannot just pull it the latest CMSIS from ARM. Otherwise - the system works :)

 

[JKapp.1]

Okay, understood. I think ST went down a suboptimal way. I don't understand why they had to restructure everything in their repos? I mean, there are the official repos by ARM and there's a package manager by ARM, so why re-inventing everything and making it more cumbersome?

Git submodules are also no proper way for package management... and the problem is not the repository, we do have Artifactory and GitLab, the problem is the functionality in CubeIDE/CubeMX.

I also do not agree to the "ST makes chips" argument. Yes of course they do make chips. But they also offer a massive software landscape, re-inventing the wheel, and pushing that "solutions" a lot. 

>So back to STM32: one can pull the STM32 libraries from github as submodules, and make the app project link/refer to these libraries. They won't pollute the user app project. But there is  a prominent warning in the "HAL" library documentation about dependency on certain CMSIS version! You cannot just pull it the latest CMSIS from ARM. Otherwise - the system works :)

And that exactly is the key argument for a package manager. If the HAL v21.4.2 depends on CMSIS v1.0.0, then a package manager must exactly solve this, download the appropriate packages, and link them/add them to the project/solution.
E.g., if I choose a STM32U5xxx and CubeMX support in Keil's package manager, it tells me that CMSIS-Core is required:

I was hoping for something similar in ST's solution.

Looks like I will go and use the CMSIS-Toolbox way, see CMSIS-Toolbox and Configure STM32 Devices with STM32CubeMX - CMSIS-Toolbox. Which is open-source and (silicon-) vendor independent BTW. 

 

 

[JKapp.1]

This has nothing to do with a debugger. I think we're not discussing the same thing here.

From my PoV STM32CubeIde is not suited for scaleable, professional, CI/CD driven development. And that's fine, just wanted to make sure.

It might be useful as an IDE for debugging, but apparently not for package/project/configuration management.

[oxc07w2]

There are different ways of realization with different advantages and disadvantages. Plus if you use one way, you are used to do it this way. To give you a different perspective from a hardware and software developer:

"Hundrets of MB" is not a reason for me, since we are no longer in the 90s. Any system update takes up more memory than library. The reference manual alone takes about 30...50 MB. I prefer efficency, but this is not the place where a battle is won, not today.

For me it is important to be as independent as possible. Exaggerated: If the world gets nuked, there is no internet any more, I lost my system, but I found my backup, then I will be able to restore the system and build the binary. For this it is helpful to have a few zip files containing all libraries I need and a way to import them. Downloading everything slice-by-slice (automatically) is disadvantageous for this.

---

@JKapp.1 wrote:

The key advantage was that there's a global config file to define all dependencies (like CMSIS-Core vxxx, or BSP, etc) in a certain version and then the package manager downloads these packages on demand. There's no need to check in all libraries/dependencies into the source control or add them to the source tree at all - they are just linked automatically.

---

In contrast, I don't like it when everything is done automatically, because that often leads to me not noticing/understanding what exactly is happening. I want to know, what dependencies exist.

There are also other ways. Another vendor of MCUs provides one huge file containing everything for every MCU on the market, including all examples and all thirt party libs. You need to copy or link everything you need. Huge, but not bad in my opinion.

[JKapp.1]

Of course there are always multiple ways. But some do follow best-practices and some do not. Or something in between. And no-one says that the best-practice way is the easiest on short term, but likely the better one on mid-term ;)

Hundrets of MB is always a reason to avoid when it comes to version control and CI/CD. 5 MB are instantly cloned. 300 MB take a minute or two. Now do this 100 times a day per project (yes sure there's some caching which might help), but do it for 10 projects - it's just not efficient. 

The reference manual does not live in the SCM btw.

It's also about separation of concerns - if I clone a user application I expect only user code to be in that, no libraries.

I get the point about nuking the world, but then instead of pulling out my hard disk backup, I pull up the GitLab backup and have everything working out of the box, assuming I've created a docker image for the CI/CD before, which is highly desired to have reproducible builds anyways. I don't download my packages from the internet all the time - just once. Afterwards they live in the Docker image or in my local package registry (GitLab or Artifactory). But they don't belong to my code repo.

It feels really weird to discuss about the advantages of package management. I would be so laughed at if I started to tell the Python, Go, JavaScript, Julia, ... people to ship their applications with all required libs. So why is that so different in C?

Again - I do understand the simplicity of having everything zipped in the same repo. Fine. But this is not how modern SW development works at scale. For home use - I don't care and would do it however the vendor's example work, that's fine. But not in a professional context.

A random find, from 2022: 3 Tips for Embedded Software Configuration ManagementBeningo Embedded Group

Edit: missed that one:

> In contrast, I don't like it when everything is done automatically, because that often leads to me not noticing/understanding what exactly is happening. I want to know, what dependencies exist.

Exactly! And instead of looking at some random folders in a repo, why not looking at one single dependency file? How do you think do dependency/vulnerability scanners like Dependency list | GitLab work?

[oxc07w2]

Maybe we are talking at cross purposes. You don't need to copy the libraries into your project and you don't need to download the libraries for every project. If you start a project for an MCU you never used before, CubeIDE downloads all libraries for this MCU, put in into a local "repository" and you have to link (not copy) all the files you need, referenced to the library folder. You can add the library folder to the search path list and simple include the files. When starting a new project for the same MCU family, all libraries are already there.

The backup of a project only contains the user code, not the libraries, but the references to the library folders. After restoring the backup you have to check if the libraries are present in the right version.

---

@JKapp.1 wrote:

It feels really weird to discuss about the advantages of package management. I would be so laughed at if I started to tell the Python, Go, JavaScript, Julia, ... people to ship their applications with all required libs. So why is that so different in C?

---

It does not depend on the language, but on the complexity of the software, the performance of the hardware and where it is used. Writing in C allows you to hardware-related developement and getting efficient code at the price of more workload (MCU). Writing in python or java allows you to write complex software with less workload at the price of the performance of the result (PC, smartphone or big MCU).

If we look at a company whose only product is isolated software, then you will probably be right. But companies that uses MCUs not only develope software, they have to develope PCBs, cases, they have to think about packaging, procurement of components and so on. Developement of simple software is only a small part and factores you have listed doesn't really matter, especially if the software developer does not only writes code. The IDE must work for those people. For bigger companies, where tens or hundret of people only write code, there are other solutions like Keil MDK. As a result, there is a solution for hobyiests up to big international companies. If there is no CubeIDE, there will be no solution for hobbyists up to medium sized companies. There are other manufacturers of MCUs with higher entry barriers and they are less successful on the market or cover other markets.

[JKapp.1]

---

@oxc07w2 wrote:

Maybe we are talking at cross purposes. You don't need to copy the libraries into your project and you don't need to download the libraries for every project. If you start a project for an MCU you never used before, CubeIDE downloads all libraries for this MCU, put in into a local "repository" and you have to link (not copy) all the files you need, referenced to the library folder. You can add the library folder to the search path list and simple include the files. When starting a new project for the same MCU family, all libraries are already there.

 

The backup of a project only contains the user code, not the libraries, but the references to the library folders. After restoring the backup you have to check if the libraries are present in the right version.

---

Yes, I know the local repository, and it's the right approach imo. My mistake was that I started with ST examples, e.g. STMicroelectronics/fp-sns-datalog2: The FP-SNS-DATALOG2 function pack represents an evolution of FP-SNS-DATALOG1 and provides a comprehensive solution for saving data from any combination of sensors and microphones configured up to the maximum sampling rate. Please check st.com where a more recent version of the software might be available., where the linking mechanism is not used, but everything is indeed copied to the repo which lead to this thread. Other examples look similarly. They are not utilizing the package installer and the local lib repository. They copy everything into the project and thus are not using the mechanism you're describing.

I get now that these are examples and PoCs, that's fine to quick start. But they do not follow best practices of SW development and thus might lead to the erroneous assumption that this is the way to structure projects in the ST ecosystem.

The other gap I'm seeing is that there's no automatic way to link the packages from the local repository automatically to the CubeIde project, keeping dependencies and fixed version numbers into account, guess that's the whole point here.

Thanks all for keeping answering me!

[Pavel A.]

So we have two almost unrelated discussions here:

(1) You acknowledge that CubeIDE does not come with a decent package manager like ARM (Keil) has. That's fine if we compare their price tags and target audience. So much about CubeIDE.

CubeIDE itself is just Eclipse CDT. It will happily build, edit and debug a suitable Eclipse project, generated by CubeMX or whatever else. CubeIDE comes with some 'team' components inherited from Eclipse. One can use these or use whatever they like.

(2) Are poor CubeIDE users doomed to keep everything in one version control repo? No they do not. CubeMX (standalone or CubeIDE internal) and its Software Pack manager does not help creating well structured projects - but other means exist for this. Professionals know their tools. 

For every STM32 MCU and eval. board, ST offers a consistent package that lets quickly make tests or POC applications. Not more, not less. It is good for users, good for marketing, good for support, mostly free of 3rd party licensing troubles. Is this also good for real application development? Of course it is not. ST does not want to compete with their software ecosystem partners.  That's fine.

 

[JKapp.1]

Yes I think that's a good summary. The examples by ST are just examples or PoCs. That's it. I think that's a good "mind-set", but people I spoke to recently (in my company) take these for best practice examples, perhaps that's where all this discussion comes from.