I'm starting to put down some ideas for the high level interface to urbansim, you can see some sketches here: https://gist.github.com/jiffyclub/0bb253757547fbc14add. There isn't a huge difference between the three sketches there now, but I'll keep thinking about things and probably our dialogue will spur some ideas all around.

Let me know what looks good, what's missing, what's on your wishlist for this, etc.

It's kind of limited sampling, not something you could use during a location choice model. Alternatives may show up as available for more than one chooser and alternatives may show up more than once for a single chooser.

I cant get conda to install urbansim > 1.3, Win 10, Py 2.7 64 bit

IPythonNotebookScratch>conda config --add channels synthicity
Skipping channels: synthicity, item already exists
IPythonNotebookScratch>conda install urbansim
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: ................................
Package plan for installation in environment C:\Anaconda2:

The following NEW packages will be INSTALLED:

    urbansim: 1.3-py27_0

Proceed ([y]/n)?n
IPythonNotebookScratch>conda install -c https://conda.anaconda.org/synthicity urbansim
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ........
Solving package specifications: ................................
Package plan for installation in environment C:\Anaconda2:

The following NEW packages will be INSTALLED:

    urbansim: 1.3-py27_0

Proceed ([y]/n)?n

What am I doing wrong here?

@fscottfoti, this implements the supply/demand comparison and adjustment we'd specced out. Very simple, straight comparison of (probabilities * len(choosers)).groupby(alternatives[col].values).sum() / alternatives[col].value_counts() with adjustments done across 5 iterations, capped at 0.75-1.25 fractional change per iteration.

Made some changes to the transition model. These include:

1. Update _updated_linked_table method to use pandas merge:

I was hitting a performance bottle-neck when adding a large number of agents (and therefore linked agents). I was adding ~70k agents and 194K linked agents and this was taking around ~34 seconds. Changing the code to use the merge reduced this to ~4 seconds.

2. Add ability to control to a count provided by an 'accounting' column:

This was a feature that MAG had added to the old opus code. For example, we may want to sample from households but the control we want to hit exactly is household population. I added a new sampling file to the utilities folder to handle most of the heavy lifting w/ the idea that maybe this functionality could be of use outside the transition model as well. Changes were then made to add and remove rows methods and each of the transition classes to accommodate this.

I've tested this, and everything seems to be working fine. But I get that this is addressing pretty core functionality so if you are not comfortable bringing this in right now I'm cool w/ mirroring the transition file (something like mag_transition?). We could also leave in this in our specific urbansim implementation but I thought it might be of use to other agencies.

This addresses feedback when the accounting total is not met (#178) and also providing probability distributions to influence the sample (#149).

To obtain feedback, set the return_status argument to True, and this will return both the sampled rows and the status. If False, then only the sampled rows will be returned. This is still the default, so as to not break any existing calls.

To provide sampling probabilities, set the prob_column argument to the name of the column in the data frame containing the weights or probabilities. These values will be normalized so they sum to 1, if they do not already.

The logic for sampling with accounting but without replacement has been changed to hopefully be more robust.

Function utils.sampling.sample rows includes a loop which shall converge to an exact pick of samples. In some cases, this loop cannot converge. Example:

import pandas as pd
import urbansim.utils.sampling
df = pd.DataFrame({'tot': [2,2,2]})
sample = urbansim.utils.sampling.sample_rows(total=3, data=df, replace=False, accounting_columns='tot')
sample.sum()  # = 2 and not = 3 as expected

Even if an exact pick is possible, the random permutation l.51 causes a further problem. If the values needed for an exact pick are at the head of the randomized list sample_idx, they are not available in the loop to adjust for an inexact pick, the loop cannot converge. This aspect depends on the random seed, so it can make integration tests fail randomly.

Suggested solution: there should be some feedback from the sample_rows function whenever convergence to an exact pick is not achieved. The caller may use this information if he wishes. In this way a more robust test suite can be obtained.

This adds a cache_scope option to the table, column, and injectable decorators that allow results to be automatically cleared from the cache at set points. The options are 'step', 'iter', and 'forever'. Items with scope 'step' are cleared after every model is run, items with scope 'iter' are cleared at the end of every model year, and things with scope 'forever' are never automatically cleared.

This removes the need for table_source so that has been removed. The equivalent is now to use cache_scope='forever'.

I waffled a little bit on using 'forever' or 'simulation' for the long-lived scope. With a scope of 'simulation' it'd make sense to clear it at the end of the simulation, with 'forever' it'd be best to never clear it. I'm up for either one, anyone have an opinion?

This enables something like this:

@sim.injectable(autocall=False, memoize=True)
def my_utility_func(x, y, z):
    return x + y + z

@sim.column()
def my_col(another_col, my_utility_func):
    result = my_utility_func(1, 2, 3)
    return another_col * result

And if you call my_utility_func again with the input of 1, 2, 3 you'll get back the cached result instead of re-doing the calculation.

Memoization requires autocall=False and that function inputs always be hashable. The memoization cache follows the same rules as other caching, so you can use cache_scope=, sim.clear_cache, sim.disable_cache, and sim.enable_cache as usual.

For tracking money. Right now it has .add_transaction(amount, subaccount, metadata), .add_transactions(transactions), and .to_frame() methods. You can access the balance or list of transactions via the .balance and .transactions attributes, respectively. What else does it need?

• return pd.DataFrame with pd.Int64Index() index in remove_rows()
• add assert_empty_int64index() to use with test_tgrtransition_with_accounting()

I tried changing as little code as possible.

Note that the failed build is due to changes in pandas (possibly 0.18 (or >=0.17.1)), as commented here.

Alternatively, for the test_*remove*all() tests at least, the check_index_type option in pdt.assert_frame_equal() can be set to False. This isn't guaranteed to fix everything.

In Pandas 1.2+, pandas.Index.to_native_types() is deprecated, raising warnings like the following:

This comes up in code that serializes data to yaml for storage and later reloading.

urbansim/utils/yamlio.py#L48

The replacement suggested in the message doesn't sound as general-purpose, but maybe it would work if Pandas is able to convert string representations of ints and floats back to the appropriate data type. Another option could be to use to_json().

pandas.Index.to_native_types() pandas.Series.astype() pandas.Series.to_json()

Hey guys, I'm just going through this code and noticing, it doesn't seem to have the same functionality as what's been implemented in the UrbanSim Cloud Platform. Where can I find the files being used in the UrbanSim Cloud Platform? Are they also open source?

Also, I have found an old version of the SqFtProForma.py file and it seems to have more functionality than the file in this repo. Was this file not working? Is this urbansim repo old?

Old file was here https://github.com/UDST/developer/blob/master/developer/sqftproforma.py

Description of the bug

Wes McKinney's intro to pandas is no more available http://udst.github.io/urbansim/gettingstarted.html#pandas

Vimeo link points to nowhere https://vimeo.com/59324550

Some of our unit test syntax has been deprecated and removed in recent versions of PyTest.

For example, tests that directly called the df() fixture defined in test_mnl.py#L82 were raising errors when I worked on PR #222.

This shouldn't be too hard to fix, but will require going through and setting up some of the test data in alternative ways. It doesn't seem especially high-priority, though.

For now, I've pinned pytest at v3.10 in the Travis and AppVeyor scripts.

When running transition model, the resulting data frame loses the index name (I'm guessing because of the concat). Is this worth preserving? This would be a simple fix, but not sure if it would break existing implementations.

The transition model will not work well if the subset of agents to be sampled from is very small or even empty. Thus, an option of relaxing the filtering conditions (as present in Opus) would be very useful (e.g. if there are not enough households of the given characteristics within the given city, sample from the corresponding county or the whole region).

If nothing else, an option to use a user-defined callback function would help, together with letting the model to access the model configuration. It would replace/extend the filter_table call: https://github.com/UDST/urbansim/blob/79f815a6503e109f50be270cee92d0f4a34f49ef/urbansim/models/transition.py#L305