Updated tests

[Jfeatherstone]

This pull request expands the tests for the library to cover more use cases, as well as to be more helpful in debugging. The motivation behind this is to make it easier to review #115. Sorry it took so long since we first discussed it; the current version of library fails several of the new tests, and I wanted to make sure that I had an explanation (and a fix in #115) for each one.

There is still a focus on power laws in the tests, but now every distribution is tested for both continuous and discrete data, with and without xmax values, and for generating random numbers. The testing functions have also been split across multiple files for easier maintenance:

• test_reference_data.py: Mostly unchanged from test_powerlaw.py though the data loading was changed to allow the tests to be run from any directory; this tests the fitting methods on real datasets.
• test_plotting.py: Unchanged from previous version, just tests using custom bins in plotting. Should probably be expanded in the future.
• test_pickle.py: Changed from previous version, though still tests whether the powerlaw objects (Distribution and Fit) can be pickled. Instead of using real data, generates synthetic data. Also, the old version used a custom "walk" function to test equality between two Fit objects; this is not ideal because trying to fix or maintain that specific (rather technical) function seems like extra, unnecessary work when we could just use more basic techniques to make sure the objects are the same.
• test_generating_data.py: New file, tests random number generation from distributions, including with upper/lower bounds, and for discrete/continuous data
• test_synthetic_data.py: New file, tests generating random data from a distribution with fixed parameters, fitting that data with a Fit object, generating more data from a subdistribution of Fit object, and then fitting again. Not ideal for identifying specific issues because it is such a complex task, but this encapsulates the typical usage of the whole library. Performs this process for all available distributions, for both continuous and discrete data, and for with and without xmax values.
• test_powerlaw.py: New file (though there used to be a file with the same name), focuses on fitting externally generated power law data to help isolate issues with internal random generation vs fitting for power laws.
• test_lognormal.py: New file, focuses on fitting externally generated lognormal data (using a multiplicative random walk) to help isolate issues with internal random generation vs fitting for lognormals.
• accuracy_benchmark.py: New file, not a unit test, but a script that performs fits on generated data to gauge the typical level of error for fitting various distributions. This is useful because passing the tests might tell us whether the fitting is working overall or not, but it can't identify if the fitting became less or more accurate.

Some other notes:

• I think it is important to have test cases for each distribution like test_lognormal.py, where the sample data and the true parameter values are generated each time, not just saved from a previous software/version. This is a little more difficult than using cached data or results from another library since it involves finding a process or way to generate data where you know the true parameters of the distribution. The major advantage here is that we aren't depending on another library or version of code (which might have its own bugs) and we also aren't depending on the internal RNG, so we can isolate where a bug might be located. Eventually, it would be good to have such cases for every implemented distribution.
• There were some R files in the testing directory that presumably were used to compare this library to another implementation of similar methods. These files seem to comprise an entire package in themselves (maybe a very old version of poweRlaw?), so I don't think they should just be sitting in a testing folder of another repository, especially since the test_reference_data.py file already compares results to (cached) results from another library. I've removed these files; please let me know if you'd rather keep them around, and I can restore them.
• The tests for testing discrete truncated power law distributions take several minutes to run, primarily because we have to do a bisect search for each random value on the function ccdf() which are quite heavier than other ccdf() functions since they involve inverse gamma functions. No other issues here, just expect them to take several minutes each.

With that in mind, to run the tests, you should now use the command below to make sure every file gets run (I've updated the Github CI to do this too).

python -m pytest testing -v

The current version of powerlaw fails several of these new tests; I believe this is due to bugs in powerlaw, and not due to issues with the new tests. Below is an explanation for each test that fails.

• testing/test_generating_data.py::TestRandomGeneration::test_rng_bounds_continuous
testing/test_generating_data.py::TestRandomGeneration::test_rng_bounds_discrete
test_generating_data.py::TestRandomGeneration::test_rng_lower_bound_power_law_alpha_0p5_to_0p9
  There are some issues for the bounds of random numbers generated from distributions. The upper bound test fails because the current random number generation doesn't respect the distribution's xmax values. This should be the expected behavior of a random number generator based on a distribution that has explicit bounds on the domain. test_rng_bounds_discrete also fails for another reason, which is discussed with the other discrete issues below.

• test_generating_data.py::TestRandomGeneration::test_rng_upper_bound_power_law_alpha_0p5_to_0p9
  Related to above , but now it is the lower bound test that fails for power laws with exponents less than 1. This is a different issue: the xmin value will be the maximum of the random samples, and all values will be drawn less than this, giving very small numbers (~1e-10). Not totally sure why, this likely has something to do with the limits of the uniform random numbers that are input into the inverse sampling transform.

• test_powerlaw.py::TestExternalPowerLaw::test_alpha_1p0_to_1p5_continuous
  This test fails because of the discontinuity for power laws with an exponent close to 1. The estimator defined in powerlaw.Power_Law.generate_initial_parameters() (sourced from Clauset et al. (2009), Eq. 3.1) fails in this region, giving values around ~1.15 for distributions with a true exponent close to 1. Since the wrong value ~1.15 is in the current valid region of power laws (1 to 3) , numerical optimization isn't performed, which would otherwise fix this issue.

• test_synthetic_data.py::TestGenerationFitting_Exponential::test_exponential_discrete_xmax
  This test fails because there are some issues with discrete random number generation, especially when you set an xmax value. I believe this isn't a problem specific to exponentials, as I have seen this error come up in other cases too, but I don't know how a good way to reproduce it for all distributions. test_rng_bounds_discrete fails because of this same issue for the exponential distribution. The actual runtime error is zero-size array to reduction operation minimum which has no identity, but it is caused by Distribution._double_search_discrete() attempting to call Distribution.ccdf(data=[x2]) when x2 is outside of the domain of the function.

#115 (after some changes; see there for more information) currently passes all of these new tests. Note that for running these tests on v1.6.0+ you need to change a few lines in test_synthetic_data.py and test_lognormal.py due to changes in how parameters are stored in distributions. These are all marked by a "TODO" flag (just Ctrl+F "TODO" if your IDE doesn't highlight them automatically) so you can easily swap them.

[keflavich]

@Jfeatherstone this is a huge amount of work, and well-described. Thanks for all of this. I'll comment in #115 more.

[keflavich]

To get tests passing here, I'd like to proceed by marking each failing test with 'xfail' and pointing to a (new) open issue.

[Jfeatherstone]

Previously, I only added xfail annotations to the test_exponential_discrete_xmax test, though I'm going to add them to all of the *_discrete_xmax tests since one of the previous CI tests failed for the lognormal distribution (see Tests/test-macos (3.11) for more info).

As discussed in the PR description and in #120, the relevant bug applies to all distributions, though typically we only see it with the exponential, and it is hard to reproduce for other, longer-tailed distributions. Probably most of the time, these tests will pass, but, as above, they may fail for the exact same reason test_exponential_discrete_xmax almost always fails.