Hey, I’m making an attempt to expand Arch’s doxygen documentation, and I noticed that all of the class names are prefixed with an underscore.
This will result in none of the commands having documentation generated for them, which is a major gap in the documentation. Underscore prefixed variables/classes/functions are excluded per the web documentation doxygen config file’s EXCLUDE_SYMBOLS variable. (See src/Doc/BuildWebDoc.cfg.in)
I wanted to know why this was done in the Arch workbench. Is it purely a stylistic choice? Would Arch’s maintainers be willing to part with the underscores? Removing the underscore from the doxygen configuration file may or may not break something else in the documentation, I have yet to figure out why it’s in there.
Using the underscore was a mistake, of course. The main author of the workbench probably didn’t anticipate that it would cause problems in documentation when he initially started the workbench back in the distant past.
We could of course change the name of the classes, but this would break the loading of older files. When a file is loaded, each object is recreated by loading its class. So, if you have an Arch_Wall, it will try to reload ArchWall._Wall. If it cannot be found, the geometry will load, but as a static object, no longer as a fully parametric object.
So, if you want to rename the classes, you need to set up an alias so that the file loads the object it is looking for.
# ArchWall.py
class Wall
...
_Wall = Wall
Once that is done, then you need to be sure the object can be opened and works correctly. This is the reason I suggest creating unit tests.
I suggest writing a test file that has all Arch objects. This is a sample file that you create and can open with the current master version. Then as you make modifications to Arch, you need to try to open the sample file. If all Arch objects open correctly, it means the migration from the old class to the new class was successful, and there won’t be broken files.
See Draft/drafttests/draft_test_objects.py for a sample file for Draft. We would like to have something similar for Arch.
Another possibility is to write code that upgrades an object from an older class to a newer class. It would basically recreate the object, using the new class name. It would iterate over all the older object’s properties, and assign them in the new object.
# old object is loaded
old_obj = App.ActiveDocument.OldWall
# New object is created
obj = App.ActiveDocument.addObject("Part::FeaturePython", "Wall")
Wall(obj) # New class
# iteration over all properties of old_obj
obj.PropertySomething = old_obj.PropertySomething
obj.Shape = old_obj.Shape
Check all properties inside obj.PropertiesList. The same process for the view provider classes.
That is quite the pickle. I’ll look into the unit tests. I’ve also noticed a few other strange things about how the documentation is generated so this may be an appropriate time to fix several things.
That sounds good. See source documentation and Doxygen. Doxygen is pretty powerful but it’s a bit hard to get everything right. It particular it’s hard to make live tests with FreeCAD because even small changes to the code results in big recompilation times of the documentation.
I had one issue, maybe you can look into it. Once you generate the Doxygen documentation, in the HTML pages there are some arrows that expand subtopics, for example, C++ classes show which other C++ classes they inherit. In my computer these arrows don’t work. It works in the online API pages, but not locally.
And also, I think a task that should receive priority is investigating a way to handle both C++ and Python in the best way possible to create good documentation. Since FreeCAD is a combination of C++ code with Python, it’s actually a bit hard to manage both languages in a simple way.
One way of handling things is using Doxygen completely. It would parse C++ without issues, but then it would use the program “doxypypy” to process Python sources, so that the generated documentation looks better. I investigated this briefly and it seems to work, but I didn’t test on a wide scale, just in my computer. See Doxygen.
The other path is using Sphinx to handle everything. It would parse the Python modules natively, and when it comes to the C++ code, you’d use an intermediate program called “Breathe” to call Doxygen in order to parse the C++ code.
If you can find clever ways on how other projects handle this combination C+±Python, that would be great.
I’ve found two solutions to documenting mixed python and C++.
1) Stick with doxygen
So after some experimentation, I’ve discovered the most elegant solution I could find, which isn’t very elegant. An undocumented feature of doxygen is that if you add an exclamation mark at the start of a docstring, doxygen will render the docstring appropriately. By appropriately, I mean showing line breaks, and rendering markup formatting.
Upsides: We get to keep using doxygen without any changes.
Downsides: All the docstrings have an exclamation mark at the start of them, which might confuse people using help(), or an autocomplete. Also, we’re stuck with all the current things we dislike about doxygen.
2) Move to sphinx
I believe this would produce higher quality documentation overall. I’ve never used sphinx in this particular use case, but personally, I find it much friendlier as both a user and a developer. I think this is the way to go.
Upsides:
Documentation generated quickly and easily. Lowers the bar to entry for those who want to document things.
Greater flexibility in how the documentation is structured, instead of sticking to doxygen’s rather c++ oriented structure. Currently, I find the api doc’s structure rather arbitrary. I think we can make it much more user friendly.
Good looking.
I could just implement this for Arch, and we can see how that turns out, and expand it if it’s acceptable.
Downsides:
This turns this into a larger project.
Sphinx being more flexible could down the road, end up causing this documentation to start edging into the wiki’s territory.
I’m happy to pursue either option, on your recommendation.
Since we have never truly explored Sphinx, I suggest looking into this more. A long time ago the developers did create some tentative documents, but the efforts were not pursued further. You can see the few files tested in src/Doc/sphinx.
In C++ leading underscore names are used to prevent name collisions between “implementation” names and application names. IIRC, application programmers are not supposed to use leading underscores.
Alrighty, I’ll see if I can produce something that works better than the current system, and report back when I’ve got a better idea of it’s limitations.
Ok, I’m pretty certain that sphinx is the way to go.
I’ve produced some sphinx documentation that can happily extract stuff from both python and c++, as you can see here:
It’s also got some scripts that will automatically regenerate the documentation when you make changes to things. The key point here is that these scripts will only regenerate the specific parts of the documentation that need to be changed, to keep down the doc’s build time. it is currently far faster than doxygen.
For documenting c++, you keep using doxygen documentation syntax. For documenting python, you use the google docstring styleguide, as per here, with examples here. This keeps the doc strings human readable, so they’re suitable for reading straight from the code, or from the FreeCAD python interpreter’s autosuggestions. It also allows sphinx to extract syntax, as you’d expect from auto-generated api documentation. I think this is a very elegant solution, that will not effect anyone who wants to keep using pure doxygen.
Sphinx gives us the control to choose what’s included in the API documentation and not, so we don’t end up with a voluminous tome of auto-generated documentation without any useful content.
The downside is that it’s a little more complex than I’d like, and that as far as I can tell, it’ll only work on linux and mac systems. It works very nicely once you’ve got all the auto generation terminals open though.
About C++, it’s not only about documenting a single class or function, but rather automatically generating the relationships between them and creating the diagrams that show the relationship between the classes. That is very helpful when using Doxygen.
For documenting c++, you keep using doxygen documentation syntax. For documenting python, you use the google docstring styleguide,…
Can you check if we can use Numpy style for the docstrings? I’m not against Google style on principle, however, I think since FreeCAD is a technical program that deals with mathematics, I feel that it’s more appropriate to use the same style as Numpy, SciPy, and Matplotlib. This is what I’ve been doing since I started adding docstrings to various files. See for example, WorkingPlane.py.
I normally use Spyder to write Python code. Spyder is an IDE primarily meant for scientific use, so by default it parses docstrings in Numpy style and displays them nicely. I actually encourage people to use Spyder every time I can, but it seems to me not many people use it, probably only those who use Numpy and Matplotlib regularly.
Sphinx gives us the control to choose what’s included in the API documentation and not, so we don’t end up with a voluminous tome of auto-generated documentation without any useful content.
Ideally we would specify the Python workbenches to run through Sphinx, and the rest of the C++ workbenches to run through Doxygen. The problem is you can also include Python commands in C++ workbenches without much problem. For example, see src/Mod/PartDesign/WizardShaft, FeatureHole, InvoluteGearFeature, etc., these are Gui Commands and classes to create scripted objects in Python. So, how to tie the two worlds would be interesting.
The downside is that it’s a little more complex than I’d like, and that as far as I can tell, it’ll only work on linux and mac systems. It works very nicely once you’ve got all the auto generation terminals open though.
Ideally we can run the documentation generation script through “make DevDoc”, just like we do right now. Or how do you intend to generate it? Also, why would it only work in Linux and MacOS?
Do you mean you are running some scripts with Bash? Could you write an equivalent .bat file, or powershell script to run the right commands in Windows? This should be set through the initial cmake. If we are on Windows cmake should set the proper script to run when doing “make DevDoc”.
Or if you want something more general, we could use Python itself as a scripting language to run the necessary programs. We can use subprocess.Popen to run external programs with it.
Or do you mean Sphinx itself doesn’t quite work on Windows?
I see. The system should be able to see links between classes and what not. I will confirm to see if this is actually the case when I wake up tomorrow.
We can. It works interchangeably with google and numpy style. I just picked google, because it seemed like google would have less boilerplate, but I’ll use whatever the standard is.
Very interesting. I’ll look into these examples. I have a horrible feeling the answer is: “By manually writing documentation, at great expense to sanity.”
Currently it all works through bash scripts. All the bash scripts can probably just as easily be .bat scripts.
The way it currently builds is the following process:
FreeCAD gets built.
Doxygen scrapes the c++ code, and spits out XML that sphinx reads.
Sphinx imports FreeCAD and FreeCADGui, and thus has access to all the docstrings that the FreeCAD python interpreter has access to, and best mimics the Python environment at runtime.
Sphinx builds the documentation.
The part that I believe won’t work on windows is entr. Entr is just the part that allows for the automatic rebuilding. If you don’t have it, the only consequence is you have to run the build manually after you make a change. I wouldn’t be surprised if a window’s equivalent exists, however. I just think it’s important, as I think automatic rebuilding is an important part of creating any graphical product.
Did you try with this “Breathe” program? I sort of read that it kinda did that, it used Doxygen to extract the data from C++ sources as XML, instead of creating the HTML pages.
Also, notice that in the C++ part we often have the same function or class twice, one for use in C++ and another one that is converted into Python.
For example, the commands of the Part workbench are described in various files, but the documentation generated with Doxygen only really tells you about the C++ structure. Take the “Part.Circle” class. The Python interface is actually implemented in two files “CirclePy.xml” and “CirclePyImp.cpp”. If you run just Doxygen, it tells you about the “CirclePy” class in C++, but the actual usage of the “Part.Circle” command from Python is not described. The information is there, it’s just presented in a very technical C++ way, not in a Python way.
To see the Python information you have to run FreeCAD’s own Python documentation server, Std_PythonHelp. As I understand it, it basically runs “pydoc” to extract docstrings from the compiled Python modules, and from the actual .py files in the installed system. It’s like launching a mini-Sphinx server, right from FreeCAD.
If you run Std_PythonHelp it will launch the documentation server in a browser page, in localhost:7465, and it will show you the documentation of all Python modules in the system. This is pretty much what we want with Sphinx as well, but we want it to look pretty, of course.
The part that I believe won’t work on windows is > entr> . Entr is just the part that allows for the automatic rebuilding. If you don’t have it, the only consequence is you have to run the build manually after you make a change. …
I am not sure why we want automatic rebuilding of the documentation. I mean, we don’t automatically rebuild the FreeCAD program itself. I’m not sure I understand the idea behind this. In my opinion, the documentation has to be built manually whenever the user wants. Issuing “make DevDoc” should be sufficient for this.
I followed up on this, and yes, sphinx will reference and link the various relationships between things. Here are some sphinx/breathe c++ documentation examples that are doing this:
I’m happy to say that I’ve just checked, and the sphinx can access these c++ based python functions, and read them as python. Here I’ve gotten both the Part.Circle class, and the function.
Eh, call it personal preference. If people just want a single command they can run that builds the entire documentation, that also exists and works.
What about a diagram produced with GraphViz? Doxygen creates a diagram of the relationship between the classes. It uses the “dot” program which is included with GraphViz.
Also, since you are progressing, maybe you can write the process and your findings in the Sphinx page on the wiki. I will create it now, and then we can fill it with information.
Well, Doxygen is providing all the XML it needs so it should work. Worst case scenario is we use the graphvis python bindings to scrape the doxygen XML ourselves. I don’t think that’ll be needed though.
def getExtrusionData(shape,sortmethod="area"):
"""If a shape has been extruded, returns the base face, and extrusion vector.
Determines if a shape appears to have been extruded from some base face, and
extruded at the normal from that base face. IE: it looks like a cuboid.
"""
Good
def getExtrusionData(shape,sortmethod="area"):
"""If a shape has been extruded, returns the base face, and extrusion vector.
Determines if a shape appears to have been extruded from some base face, and
extruded at the normal from that base face. IE: it looks like a cuboid.
"""
By the way, what the hell is this? This looks terrible.
r = FreeCAD.Rotation(
FreeCAD.Vector(0,0,1),
n
)
The original code is fine already as long as it is shorter than 80 characters. Maybe just add spaces after a comma, like PEP8 suggests.
r = FreeCAD.Rotation(FreeCAD.Vector(0, 0, 1), n)
There is no need to break parentheses into separate lines, like it were C++. In Python, you should take advantage of the automatic line continuation that parentheses and brackets provide, and align things at the same logical level.
Minimum viable product. Dumb startup industry corporate speak. I just meant I’ll wait until I’ve ironed out the very obvious problems, and people can look at a large enough portion of it, that they can appreciate what it’s supposed to look like, if expanded further.
I was doing that because the autosuggest on the python interpreter didn’t handle the indentation well. See a comparison without alignment and with alignment here: https://imgur.com/a/eNCOoaN
Now taking a second look at it, it doesn’t look that bad.
Thanks for the feedback, I’ll make the rest of those other changes too.
