python - Pandas: Converting timeseries of values to tracking changes in value -

python - Pandas: Converting timeseries of values to tracking changes in value -

i've been stumped on , - uncharacteristically - net has been little help. working in pandas sense must general problem people aiming store timeseries info efficiently.

i have lots of standard daily time series info values happen alter infrequently, this:

date value 01/02/2014 .1 01/03/2014 .1 01/04/2014 .5 01/05/2014 .5 01/06/2014 .5 01/07/2014 .1

i convert info track days value changes, above illustration should this:

date value 01/02/2014 .1 01/04/2014 .5 01/07/2014 .1

unfortunately using drop_duplicates() delete required values in case value reverts prior value (like .1 in illustration above).

use combination of shift , all:

in [98]: import io temp = """date,value 01/02/2014,.1 01/03/2014,.1 01/04/2014,.5 01/05/2014,.5 01/06/2014,.5 01/07/2014,.1""" df = pd.read_csv(io.stringio(temp)) df out[98]: date value 0 01/02/2014 0.1 1 01/03/2014 0.1 2 01/04/2014 0.5 3 01/05/2014 0.5 4 01/06/2014 0.5 5 01/07/2014 0.1 in [99]: df.loc[(df.shift() != df).all(axis=1)] out[99]: date value 0 01/02/2014 0.1 2 01/04/2014 0.5 5 01/07/2014 0.1

here compare shifted (by 1 row) dataframe original dataframe, want compare each column , utilize all , pass axis=1 accomplish this.

further breakdown, if @ df.shift() != df returns:

in [100]: df.shift() != df out[100]: date value 0 true true 1 true false 2 true true 3 true false 4 true false 5 true true

we dataframe boolean values can't utilize mask is, want check rows true utilize all:

in [101]: (df.shift() != df).all() out[101]: date true value false dtype: bool

however, default checks columns true, want check row values pass axis=1:

in [102]: (df.shift() != df).all(axis=1) out[102]: 0 true 1 false 2 true 3 false 4 false 5 true dtype: bool

we can utilize our boolean mask accomplish wanted:

in [103]: df.loc[(df.shift() != df).all(axis=1)] out[103]: date value 0 01/02/2014 0.1 2 01/04/2014 0.5 5 01/07/2014 0.1

python pandas time-series