FEM Workbench Testing - GSoC Proposal

Google Summer of Code (GSoC)
21

Thanks! BTW, do you mind adding something like: Devlog: https://ceeli.github.io/gsoc/ in to your signature?

Cool! Doxygen is a beast and we have quite a ways ahead of figuring out how we should document the FC code. Currently it’s swiss cheese when it comes to API documentaion generated from within the code and there are mixed feelings about making the source code bloated by generating documentation this way. Maybe your approach can generate some consensus and even a convention around documenting ??? :pray:

22

Good idea, I just added it :slight_smile:

Yes the documentation is very sparse at the moment. Doxygen supports in code as well as embedding external documentation. Doing everything in the source blows up the code like you said but doing everything external is messy, hard to track and it’s tedious to read the code together with the documentation. I think the best way is to combine both approaches. Documentation of members definitely belong in the source code but descriptions with examples and so on of classes, namespaces or modules could be moved out into a separate file.

Like you said the most important (and most challenging) thing is that the approach is consistently used on every part of FreeCAD. I definitely think that source documentation can’t be substituted by wiki articles and if done right it’s very easy to use (for both developer and user). Because writing tests is not really meaningful without stating what the function is supposed to to I’m going to write a lot of documentation. Lets see how my approach of documenting evolves during GSoC and maybe if it’s received well by the community we can take parts or all of it for all of FreeCAD :slight_smile:

23

I’m looking forward to this! Thank you very much @m42kus :smiley:

24

This week I wrote more C++ tests and documentation for the C++ module specifically for Fem::Constraint. While doing so I extended the C++ test support library further with useful tools. For details have a look at my devlog or the github repository. While looking through the code a noted a few things that don’t affect the working of the workbench directly but are probably worth fixing. I wrote most of those points into notes of the source documentation. In the documentation of Fem::Constraint you find a few of these notes: https://ceeli.github.io/fcdoc/classFem_1_1Constraint.html.

I’m far from completing all unit tests that should be written for the modules C++ code but despite that I will move on to my next target which are python tests without GUI. The general theme for the next month will be that I try to cover as many different aspects of testing as possible. After that I should have a very good understanding of the requirements and challenges of testing all aspects of a module and I will try to combine everything into a consistent and systematic testing approach. This means of course means that I won’t be able devote as much time to each aspect as it deserves but I think that the framework is much more important. I hope that if the framework is good enough we can slowly move to complete test coverage with the help of the whole community :slight_smile:

25

I fixed a few issues with my code on ubuntu and tested it on Ubuntu 16.04. My testing branch should now compile on those systems without problems. No additional dependencies are required. To execute the tests I wrote just execute the FreeCADTest binary. Be sure to do a recursive clone (with submodules) because I included googletest as a submodule in the 3rdParty folder. I don’t know if that is a problem for anyone. The problem is that ubuntu doesn’t ship compiled google test libraries. One could use the sources from googletest-dev instead but that would require distribution specific cmake code. Including googletest in the source should work on every platform without modifications. As an alternative to a submodule we could of course just copy the code into the repository…

As of now there is no switch to turn on or off the test compilation. This is certainly on my todo list but doesn’t have a high priority for now.

26

From my Debian packaging perspective, vendoring a copy of gtest in our source is fine, so long as there’s support to use the system gtest if it’s present. In Debian we can filter out unwanted copies of software that’s already available in the system.

27

Hey, I don’t know if debian does it differently from ubuntu but for ubuntu the recommended way of using gtest is to include it in the source. The libraries are not even available as binaries in the repositories. Do you have any experience with gtest on debian? If not I can look into it :slight_smile:

@All There is a lot of new stuff on my devlog btw: https://ceeli.github.io/gsoc/
I’m going to write a summery in here to in the next days :slight_smile:

28

Hi, just a quick status update: At the beginning of this month I progressed from writing C++ tests to writing tests for/in python. I wrote lots of documentation and tests for the femtools/femutils.py module. Right now I’m working on testing the solver framework and on a new kind of test that runs simulation and checks the results (not just if there where any errors but also if the results are what we expected e.g. displacement). I created a test support library analog to what I created for C++. You can find all of that in my testing branch: https://github.com/ceeli/FreeCAD_bhb/tree/testing.

I’m also spending a lot of time making my in source documentation appear nicely in sphinx and created a kind of sphinx prototype I’m going to share soon if you are interested. Does anyone know what the most popular opinion is for documenting python code in FreeCAD? Is it doxygen or sphinx? What I’m doing now is using sphinx with readthedocs and I combine in source documentation with pages written only for sphinx. That’s of course more work than using only API documentation generation but I believe that the end result is way better. I think there is little harm in using doxygen for C++ and sphinx for python and the benefits outweigh the drawbacks (e.g. consistency).

29

Hi, I played around with sphinx and wrote some documentation for the FEM workbench specifically the solver framework. It’s not finished but you can have a look at readthedocs: https://freecad-python-api.readthedocs.io/en/latest/fem/solver/overview.html or https://freecad-python-api.readthedocs.io/en/latest/fem/solver/settings.html. The other modules are empty, I just inserted them to show how it could look :slight_smile: My experience is that sphinx works much better with python than doxygen. Doxygen works great for C++ but I found doxygen with python almost unusable. I think that the best approach would be to use doxygen for C++ and sphinx for python. I think that is done to a certain extend already but not very consistent. Honestly I am a little confused about what the current state of documentation is in FreeCAD :stuck_out_tongue:. What do you think about that?



30

looks great!

31

@m42kus thank you so much for getting the ball rolling on this. It’s sounds a PITA to have 2 different frameworks for documenting FC (doxygen and sphinx for c++ and python respectively) but honestly it makes sense if it helps us get around the rut that we’ve been in stalling on having robust documentation for FC. I’m not sure what the technical future pitfalls may be in pursuing both frameworks at the same time. I will say though that I love the examples that you’ve created and am actually quite excited. So thank you!

32

I think a big problem (from the view of documentation) is that in FreeCAD the C++ and Python Parts are both huge. This means that we need a good efficient solution for both languages. In the C++ world people are used to using Doxygen and in the python world Sphinx is king. Also in FreeCAD it is possible to archive most programming tasks in both languages which is great but easily leads do redundant and confusing documentation. We also must not forget to consider the usability of the documentation in addition of issues with writing and maintaining it. It would be sad if we invest many hours into writing good documentation that people don’t use because the software doesn’t work well for the programming language (indexing, search and so on).

33

So true, lets really think of a strategy for this!

Yea, sphinx makes more sense for python. It does use .rst right?

34

I just finished editing some relevant pages in the wiki.

  • CompileOnUnix: I edited this page because I wanted to have clear instructions to compile FreeCAD myself. I mostly reviewed the packages needed for Debian/Ubuntu, as I use Ubuntu myself, and I tried to clarify the steps that you must follow in general in Linux for compilation.
  • Source documentation: this page explains how to create the documented sources; it follows the same process as compiling FreeCAD in general, that is, cmake and then make DevDoc, instead of just make. FreeCAD produces its online documentation in a way that I still don’t understand very well. You don’t run the doxygen executable directly, but with the cmake+make process mentioned before, so Doxygen uses the appropriate paths, variables, and configuration (like HTML headers and footers). The specifics are still a bit of a mystery, as I think the calls to cmake create temporary files (CMakeLists?), or include other files, with the right configurations.
  • Doxygen: this page was short so I expanded it with many details explaining how to use Doxygen in general, and for C++ and Python files that are used in FreeCAD.

Normally, Doxygen cannot read very well the docstrings of Python source code. However, the program doxypypy can be used to convert the docstrings to a format that Doxygen can understand better. So, when running Doxygen, doxypypy can be used as a filter, through which all Python files go, in order to pre-process the comments, and produce better documentation for Python files. This is explained in the same Doxygen page. It is a long document, but I provide examples, so hopefully it isn’t too hard to follow. At least for now I think this is the best solution to keep using Doxygen with both C++ and Python.

In the last section of the document I mention Sphinx. I wanted to do something else so I didn’t investigate further this part. It seems that, just like Doxygen is ideal for C++, Sphinx is ideal for Python. While Doxygen can process Python files with the help of doxypypy, it seems Sphinx can do the same but in the opposite direction. Sphinx can naturally process Python files, and it can document C++ files with the use of an intermediate program, like Breathe, which itself uses Doxygen. This deserves more investigation.

Clear, Functional C++ Documentation with Sphinx + Breathe + Doxygen + CMake

I’m not entirely sure what the “Domains” are in Sphinx, but the latest documentation seems to have a C++ Domain in Sphinx 3.0, which is still in development. I don’t know if this means that in the future Sphinx will be able to parse C++ natively, but it sounds promising.

Sphinx The C++ Domain

So I think there are two possibilities:

C++ and Python ----> Doxygen ----> doxypypy ----> HTML

C++ and Python ----> Sphinx ----> (natively, Breathe/Doxygen?) ----> HTML

Since I realize that you’ve made some progress on testing Sphinx, I would strongly suggest that you document the process in a Sphinx wiki page. I wanted to do this myself, but as I said, I got sidetracked doing something else. I want to refactor the Python code of the Draft Workbench (which is entirely in Python) precisely to test the Python documentation generation, as if the sources aren’t properly commented, automatic systems like Doxygen, doxypypy, and Sphinx don’t work quite well.

Personally, I think it should be possible to document everything with Sphinx. Having a nice programming interface like that of SciPy, Numpy, and Matplolib would be amazing. Or even the documentation of Blender; I believe the internals are in C++ but all public functions are in Python.

35

Yes, a consisten strategy for all of FreeCAD would do wonders. I’d gladly help to work out a good solution, is there a good place to discuss this already or do you want to make a new thread somewhere? For me the first step would be to write down the current state of the documentation. As far as I know there are a few different systems in use in different parts of FreeCAD for all kinds of purposes.

Yes it is essentially some kind of structured rst with a way of cross referencing things and a lot of plugins (e.g. for source code documentation extraction) and themes.

36

Hi, thanks for this detailed post, I definitely should go to bed now but I’ll read through all of your stuff tomorrow :slight_smile:

37

Also read this post.

38

lets split off this tangent in to another thread. Is there one already or can someone suggest a name?

39

About sphinx/ doxygen:

Some years ago I tried to use sphinx to generate the python API docs because, indeed, they look nicer. However, they fall short at the same place as Doxygen: They are unable to document C+±generated modules and classes. So at that time we made some “fake” modules (that you can still find in TemplatePyMod). that would allow sphinx to crawl through them. But obviosuly it was a PITA to maintain and got quickly obsolete.

Also, maintaining two systems is also annoying, once doxygen is able to parse python code too. So I myself basically gave up of sphinx. The code is still there but it’s not used anywhere anymore.

At some point, I think we should build some tools that automatically build fake python files from selected FreeCAD modules.. It would actually not be too hard, python has many tools to do that.

The real problem with doxygen is that the html it produces is rather ugly. But there are several other possibilities that we could try, that I saw on the net, such as generating markdown files…

40

Very nice work. Especially the compilation page would have been helpful for me when I started using FreeCAD :slight_smile: One thing I noticed is that you used triple single quotes for python docstrings. I would suggest changing that to triple double quotes because thats what PEP 257 – Docstring Convention says: “For consistency, always use “”“triple double quotes””" around docstrings. Use r"““raw triple double quotes””" if you use any backslashes in your docstrings. For Unicode docstrings, use u"““Unicode triple-quoted strings””" ". I think most projects use the double quotes for this reason.

Good Article, I especially agree with the author on the following paragraph:

On a more fundamental level, Doxygen’s style of documentation is listing out all the API entities along with their associated comments in a more digestible, searchable manner. It’s essentially paraphrasing the header files, to take a phrase from Robert Ramey[1]; embedding things like rationale, examples, notes, or swapping out auto-generated output for hand-written is not very well supported. In Sphinx however, the finer-grained control gives you the ability to write documentation which is truly geared towards getting people to learn and understand your library. – Simon Brand

Domains are just namespaces for roles and directives. E.g. with domains you can have a class directive named “class” for Python (py:class) and C++ (cpp:class). So unfortunately this doesn’t have anything to do with parsing. A parser would be a plugin like “autodoc” for python. The “Breath” project is actually a extension just like “autodoc” but it depends on the xml output of doxygen instead of the source directly. So the next step would be to replace it with a plugin that parses the C++ source itself but even that it would define new directives and roles for itself. As far as I know the C++ domain is for manually writing API documentation just like the py domain.

I would not generally say that C++ is internal and Python is public (except for scripting). The core of FreeCAD exposes C++ interfaces that are used by the workbenches and should be documented for the same reason their Python equivalents should be documented. I’d even argue that the internals of the FreeCAD core should also be documented to enable other developers to understand and enhance the core.

As I’m not aware of any other projects supporting python and doxygen I would also say that these are very good options. I still think that using both sphinx and doxygen is an option we could think about if sphinx proves to be insufficient for C++ or there is too much opposition to the idea from the community.