假设我有一个这样的数据框:
>>> i = pd.to_datetime(np.random.randint(time.time(),time.time()+10000,15),unit='ms').sort_values()
>>> df = pd.DataFrame({'A':range(15),'B':range(10,40,2),'C':range(10,55,3)},index = i)
>>> df
A B C
1970-01-19 05:31:36.629 0 10 10
1970-01-19 05:31:36.710 1 12 13
1970-01-19 05:31:37.779 2 14 16
1970-01-19 05:31:38.761 3 16 19
1970-01-19 05:31:39.520 4 18 22
1970-01-19 05:31:39.852 5 20 25
1970-01-19 05:31:39.994 6 22 28
1970-01-19 05:31:41.370 7 24 31
1970-01-19 05:31:41.667 8 26 34
1970-01-19 05:31:42.515 9 28 37
1970-01-19 05:31:42.941 10 30 40
1970-01-19 05:31:43.037 11 32 43
1970-01-19 05:31:43.253 12 34 46
1970-01-19 05:31:43.333 13 36 49
1970-01-19 05:31:44.135 14 38 52
我想要的是:
A B C
1970-01-19 05:31:37.779 2.0 14.0 16.0 #last value within 2000 milli-sec interval from 05:31:36
1970-01-19 05:31:38.761 3.0 16.0 19.0 ##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:39.994 6.0 22.0 28.0 #last value within 2000 milli-sec interval from 05:31:38
1970-01-19 05:31:39.994 6.0 22.0 28.0 *##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:41.667 8.0 26.0 34.0 #last value within 2000 milli-sec interval from 05:31:40
1970-01-19 05:31:42.515 9.0 28.0 37.0 ##last value from the ^ value within 1000 msec interval
1970-01-19 05:31:43.333 13.0 36.0 49.0 #last value within 2000 milli-sec interval from 05:31:42
1970-01-19 05:31:44.135 14.0 38.0 52.0 ##last value from the ^ value within 1000 msec interval
我可以使用以下代码实现标记有#
的行:
>>> df.resample('2000ms').ffill().dropna(axis=0)
A B C
1970-01-19 05:31:38 2.0 14.0 16.0
1970-01-19 05:31:40 6.0 22.0 28.0
1970-01-19 05:31:42 8.0 26.0 34.0
1970-01-19 05:31:44 13.0 36.0 49.0
# note I do not care about how the timestamps are getting printed,I just want the correct values.
我找不到大熊猫能提供所需输出的解决方案。我可以使用两个数据帧来做到这一点,一个在2000ms
采样,另一个在1000ms
采样,然后可能循环并相应地插入。
问题是,我的数据的实际大小确实很大,超过了4000000行和52列。如果可以避免两个df或循环,那么我肯定会接受。
注意:重复标记的*
行,因为从最后一个值开始的1000ms时间间隔内没有数据,因此将重复最后一次看到的值。同样,在2000毫秒的时间间隔内也会发生同样的情况。
如果可能的话,请给我展示一种方式...谢谢!
编辑:根据John Zwinck's comment编辑:
import datetime
def last_time(time):
time = str(time)
start_time = datetime.datetime.strptime(time[11:],'%H:%M:%S.%f')
end_time = start_time + datetime.timedelta(microseconds=1000000)
tempdf = df.between_time(*pd.to_datetime([str(start_time),str(end_time)]).time).iloc[-1]
return tempdf
df['timestamp'] = df.index
df2 = df.resample('2000ms').ffill().dropna(axis=0)
df3 = df2.apply(lambda x:last_time(x['timestamp']),axis = 1)
pd.concat([df2,df3]).sort_index(kind='merge')
这给出了:
A B C timestamp
1970-01-19 05:31:38 2.0 14.0 16.0 1970-01-19 05:31:37.779
1970-01-19 05:31:38 3.0 16.0 19.0 1970-01-19 05:31:38.761
1970-01-19 05:31:40 6.0 22.0 28.0 1970-01-19 05:31:39.994
1970-01-19 05:31:40 6.0 22.0 28.0 1970-01-19 05:31:39.994
1970-01-19 05:31:42 8.0 26.0 34.0 1970-01-19 05:31:41.667
1970-01-19 05:31:42 9.0 28.0 37.0 1970-01-19 05:31:42.515
1970-01-19 05:31:44 13.0 36.0 49.0 1970-01-19 05:31:43.333
1970-01-19 05:31:44 14.0 38.0 52.0 1970-01-19 05:31:44.135
还可以,除了申请部分需要很长时间!
为便于复制:
,A,B,C
1970-01-19 05:31:36.629,10,10
1970-01-19 05:31:36.710,1,12,13
1970-01-19 05:31:37.779,2,14,16
1970-01-19 05:31:38.761,3,16,19
1970-01-19 05:31:39.520,4,18,22
1970-01-19 05:31:39.852,5,20,25
1970-01-19 05:31:39.994,6,22,28
1970-01-19 05:31:41.370,7,24,31
1970-01-19 05:31:41.667,8,26,34
1970-01-19 05:31:42.515,9,28,37
1970-01-19 05:31:42.941,30,40
1970-01-19 05:31:43.037,11,32,43
1970-01-19 05:31:43.253,34,46
1970-01-19 05:31:43.333,13,36,49
1970-01-19 05:31:44.135,38,52