I have a test which calculates the norm of an array using numpy's `np.linalg.norm` to calculate the norm of an array and then compares it to an expected value. In my local machine the expected value is 237117.7713446387 and the test passes. However when I push it to the pipeline the calculated value is 237117.77134463872 which makes the test fail.
This definitely looks like a floating point precision problem, I checked the python and numpy versions in the pipeline and on my local machine and I believe they are the same. I really need to have exact values being compared and not use approximations with a tolerance...
I also saw that it is possible to debug pipelines locally using Docker, but this is problematic since I frequently use figures generated on the fly by matplotlib to inspect results. This is very difficult to do from within Docker as outlined by my question on SE
What is a good way to solve or circumvent this problem? I have had bad experiences in the past using approximations in my tests as slight regressions sometimes will not be caught. A test can appear to succeed when in fact it is failing.
This could look like a configuration problem, I also read your question that you sense this as well (you mention floating point precision and also using the docker container locally as well).
As this is very specific, I perhaps would add some "debug" information spilled out by the test-run, e.g.:
This might already shed some differences.
Next thing would be to make the testing with defined and clear precision settings, for me this looks like a bit that the different build systems might have slight configuration differences here (implicit). An improvement then could be to make the precision setting explicit. Check the library and/or the pyhton configuration for such settings and don't use defaults but the expected values.
A stable test-suite should be immune to that (and if you do the testing with different precision settings [e.g. the same test run multiple times with the different precision settings], you might need to skip some of the tests if the required precision value is not available).
If the library / python is not able to fully control the precision and this is bound to the underlying system architecture, then testing remains unstable unless you can handle it with tolerance (which you say you can't) or bringing the test systems en-par.
I'm guessing a lot here, hopefully this offers you some options. I'm not a numpy user, from what I could gather only very quickly, numpy is based on C binaries which then can have different precision (and maybe even floating point number handling) per installation (e.g. SE: how set numpy floating point accuracy?; NumPy: NumPy Datatypes; see Extended Precision for more options and also debug information)
...hey are a part of us, shaping how we interact with the world around us. The same holds true for programming languages when we think about how different kinds of vulnerabilities raise their heads in t...
Connect with like-minded Atlassian users at free events near you!Find an event
Connect with like-minded Atlassian users at free events near you!
Unfortunately there are no Community Events near you at the moment.Host an event
You're one step closer to meeting fellow Atlassian users at your local event. Learn more about Community Events