Try to convert dss.Properties.Value() return to numbers

[daniel-thom]

Feature Request

The return of dss.Properties.Value() is always a string. That leaves it to the higher-level application to attempt to convert the strings to a float or int. A secondary effect is that dss.utils.class_to_dataframe() also returns numeric data as strings.

Can OpenDSSDirect perform the conversion automatically so that users don't have to do it? I'm sure that you don't want to break the existing API. And perhaps you would prefer to push this to users.

[PMeira]

This will be addressed by the new low-level API. It should be merged very soon to DSS C-API.

[PMeira]

I'm working on the documentation of the new property system for dss-extensions/dss_capi#109, and probably will complement it to achieve better results for the dataframe usage. I'll post a link to the results here when I finish so we can gather feedback on it.

For a quick intro, the new system doesn't keep the string copies of the inputs anymore, so we can copy the raw data directly in most cases. I almost integrated Apache Arrow into this, but decided to avoid complicating things in Pascal even more.
As soon as we migrate from Pascal, we can start generating the dataframes (for properties, monitors, meters, reports in general, etc.) directly.

As an example of my current tests, these are the dtypes that result from building a dataframe for loads using the new Batch API:

phases                int32
bus1                 object
kV                  float64
kW                  float64
pf                  float64
model                 int32
yearly               object
daily                object
duty                 object
growth               object
conn                  int32
kvar                float64
Rneut               float64
Xneut               float64
status               object
Class                 int32
Vminpu              float64
Vmaxpu              float64
Vminnorm            float64
Vminemerg           float64
xfkVA               float64
allocationfactor    float64
kVA                 float64
%mean               float64
%stddev             float64
CVRwatts            float64
CVRvars             float64
kwh                 float64
kwhdays             float64
Cfactor             float64
CVRcurve             object
NumCust               int32
ZIPV                 object
%SeriesRL           float64
RelWeight           float64
Vlowpu              float64
puXharm             float64
XRharm              float64
spectrum             object
basefreq            float64
like                 object
dtype: object

Besides the types, a bonus is that this is around 20x faster than the current approach using the old property API.

[daniel-thom]

That looks great! Thanks.

[PMeira]

No dataframes yet, but in the latest release we have two alternatives. I believe that the next steps of development will address
this issue, #21 and #74 nicely.

JSON

import json
import opendssdirect as odd
from dss import DSSJSONFlags as flags

odd.Basic.SetActiveClass("Load")
items = json.loads(odd.ActiveClass.ToJSON(flags.Full)) 

Items is a list of dicts like:

{'DSSClass': 'Load',
 'name': '25609_a',
 'phases': 1,
 'bus1': 'ckt7.1',
 'kV': 7.2,
 'kW': 2351.25,
 'pf': 0.95,
 'model': 1,
 'yearly': '25609_a',
 'daily': None,
 'duty': None,
 'growth': None,
 'conn': 'wye',
 'kvar': 772.8185023018021,
 'Rneut': -1.0,
 'Xneut': 0.0,
 'status': 'Variable',
 'class': 1,
 'Vminpu': 0.85,
 'Vmaxpu': 1.05,
 'Vminnorm': 0.0,
 'Vminemerg': 0.0,
 'xfkVA': 0.0,
 'allocationfactor': 0.5,
 'kVA': 2475.0,
 '%mean': 50.0,
 '%stddev': 10.0,
 'CVRwatts': 1.0,
 'CVRvars': 2.0,
 'kwh': 0.0,
 'kwhdays': 30.0,
 'Cfactor': 4.0,
 'CVRcurve': None,
 'NumCust': 1,
 'ZIPV': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 '%SeriesRL': 50.0,
 'RelWeight': 1.0,
 'Vlowpu': 0.5,
 'puXharm': 0.0,
 'XRharm': 6.0,
 'spectrum': 'defaultload',
 'basefreq': 60.0,
 'enabled': True,
 'like': ''}

This may not be as fast as the other alternative I mentioned in March (using the batch operations), but should be reliable and shouldn't break.

Without flags.full, only the properties set by the user, in the order given by user, are returned for each object. There are other bit flags to tweak the behavior:

https://github.com/dss-extensions/dss_capi/blob/934f9d745a37a907749ad2bbf05ef43572507bc6/include/dss_capi.h#L6926-L6951

The last two (state and debug) will be added for a future version. The dataframe support will be similar in terms of options. Applying pandas.read_json to the ToJSON output actually gives good results in general.

New API

This is more experimental and should change a bit still. Feedback is welcome 🙂

Currently, all OpenDSS objects are exposed in DSS Python in DSS.Obj, but only the DSS properties, at the moment. The idea would be to enrich this Obj to include everything from the old APIs (both DSS Python and ODD.py) and extend freely, as we don't have to care about backwards compatibility, eventually moving it to a dedicated package. For compatibility with EPRI's version and all code that depend on the current APIs, we can keep maintaining the two older packages.

import json
from dss import DSS
obj = DSS.Obj

DSS.Text.Command = 'redirect electricdss-tst/Version8/Distrib/EPRITestCircuits/epri_dpv/J1/Master_withPV.dss'

for item in obj.PVSystem:
    print(item)

Prints:

<PVSystem.3p_existingsite1>
<PVSystem.3p_existingsite2>
<PVSystem.3p_existingsite3>
<PVSystem.3p_existingsite4>
<PVSystem.c_existing2>
<PVSystem.b_existing3>
<PVSystem.c_existing5>
<PVSystem.b_existing7>
<PVSystem.a_existing9>
<PVSystem.c_existing10>
<PVSystem.c_existing11>
<PVSystem.b_existing12>
<PVSystem.c_existing13>

We are not restricted to the "Active..." of the classic OpenDSS API, the following works fine:

l1, l2 = obj.Load[1], obj.Load[10]
print(l1.name, l2.name)

Each item and the batches expose a to_json function too.

Everything is using type annotations and enums where possible:

load = obj.Load[300]
load.conn

<Connection.wye: 0>

Type completion on IDEs seems to like that. The property help strings from OpenDSS are there too, so it's easier to know what is what:

Any property that is an object (or list of objects) can be accessed in two ways, just the name(s), or the actual object(s):

DSS.Text.Command = 'redirect electricdss-tst/Version8/Distrib/EPRITestCircuits/ckt5/Master_ckt5.dss'
load = obj.Load[100]
print([load.yearly, load.yearly_obj])

['residential', <LoadShape.residential>]

Batch operations work like:

load_batch = obj.Load.batch(phases=1)
len(load_batch) # 1381
load_batch = obj.Load.batch(re='^.*cust4$')
len(load_batch) # 65

For ints and floats, batches operate on proxies to avoid copying everything several times:
print(load_batch.kW) -> <dss.IObj.BatchFloat64ArrayProxy at 0x7f658fafa100>

Things like load_batch.kW += 0.1 and load_batch.kW *= 1.5 are passed directly to the low-level API, etc.

We implement the array protocol though, so other operations return NumPy arrays (and there's a to_array() method).

So, the dataframe functions will be basically selecting the columns from a batch and returning a proper dataframe.

[PMeira]

With the latest releases, I added some tests that better illustrate the new API, including creating the whole circuit without any .DSS text/code, creating objects one by one, or through the batch API:

• https://github.com/dss-extensions/dss_python/blob/master/tests/test_obj.py
• https://github.com/dss-extensions/dss_python/blob/master/tests/test_batch.py

It needs just a bit more work to get to a stable version, notably dss-extensions/dss-extensions#15

[PMeira]

New notebook with a few notes and examples on the updated JSON export: https://github.com/dss-extensions/dss_python/blob/master/docs/examples/JSON.ipynb