我试图通过迭代唯一值(合同编号)来添加从一个数据框列中获取的值。对于较小的迭代次数,脚本可以完美运行。但是,迭代超过1000个唯一值,会在结果数据帧中创建重复的值,从而减慢了处理速度,并花费了不必要的长时间进行处理。 我应该如何提高效率?
https://imgur.com/3obXPne-原始数据框
https://imgur.com/mEA8g6Z-新数据框中不必要的重复数据框
https://imgur.com/3i5gMoJ-在新数据帧中不必要的重复数据帧
import pandas as pd
import numpy as np
from datetime import datetime
df = pd.DataFrame([["AB1111",'2018-08-15 00:00:00','164','123','123'],["AB1111",'2018-08-15 00:03:00','564','453','126'],'2018-08-15 00:10:00','364','1231','1223'],'2018-08-15 00:01:00','575','1523'],["CD1111",'2018-08-16 00:12:00','514','341','1213'],'2018-08-15 00:02:00','1234','2018-08-16 00:05:00','124'],'2018-08-16 00:03:00','64',["EF1111",'534','121'],'2018-08-17 00:01:00','163'],'2018-08-15 00:09:00','524','129']],columns = ['contract','datetime','real_cons','solar_gen','battery_charge'])
# converting datetime column datatype to "datetime"
df['datetime'] = pd.to_datetime(df['datetime'])
#aggregation dataframe (new dataframe)
df_agg1 = pd.DataFrame()
for contract in df['contract'].unique()[:1500]:
print(contract)
df_contract = df.copy()[df['contract']==contract] # selecting each full dataframe from the main DF
df_contract.set_index('datetime',inplace=True) # set "datetime" column as an index
df_contract.sort_index(inplace=True) # sort index
df_contract = df_contract.loc['2018-8-15'] # select timeframe
# creating GB61074_cons column,which will be added to df_agg,from df_contract 'real_cons' column
df_contract[f'{contract}_con'] = df_contract['real_cons']
if df_agg1.empty:
df_agg1 = df_contract[[f'{contract}_con']] # first column
else:
df_agg1 = df_agg1.join(df_contract[f'{contract}_con']) # subsequent columns
df_agg1
如何在不创建这些不必要的重复项的情况下创建新的数据框? 是什么导致它们被创建?