在创建新的数据框时创建不必要的重复,该新的数据框通过遍历列值从另一个值获取值

我试图通过迭代唯一值(合同编号)来添加从一个数据框列中获取的值。对于较小的迭代次数,脚本可以完美运行。但是,迭代超过1000个唯一值,会在结果数据帧中创建重复的值,从而减慢了处理速度,并花费了不必要的长时间进行处理。 我应该如何提高效率?

https://imgur.com/3obXPne-原始数据框

https://imgur.com/mEA8g6Z-新数据框中不必要的重复数据框

https://imgur.com/3i5gMoJ-在新数据帧中不必要的重复数据帧

import pandas as pd
import numpy as np
from datetime import datetime

df = pd.DataFrame([["AB1111",'2018-08-15 00:00:00','164','123','123'],["AB1111",'2018-08-15 00:03:00','564','453','126'],'2018-08-15 00:10:00','364','1231','1223'],'2018-08-15 00:01:00','575','1523'],["CD1111",'2018-08-16 00:12:00','514','341','1213'],'2018-08-15 00:02:00','1234','2018-08-16 00:05:00','124'],'2018-08-16 00:03:00','64',["EF1111",'534','121'],'2018-08-17 00:01:00','163'],'2018-08-15 00:09:00','524','129']],columns = ['contract','datetime','real_cons','solar_gen','battery_charge'])


# converting datetime column datatype to "datetime"
df['datetime'] = pd.to_datetime(df['datetime']) 

#aggregation dataframe (new dataframe)
df_agg1 = pd.DataFrame()

for contract in df['contract'].unique()[:1500]:
    print(contract)
    df_contract = df.copy()[df['contract']==contract]    # selecting each full dataframe from the main DF
    df_contract.set_index('datetime',inplace=True)      # set "datetime" column as an index
    df_contract.sort_index(inplace=True)                 # sort index
    df_contract = df_contract.loc['2018-8-15']           # select timeframe       
    # creating GB61074_cons column,which will be added to df_agg,from df_contract 'real_cons' column
    df_contract[f'{contract}_con'] = df_contract['real_cons']   

    if df_agg1.empty:
        df_agg1 = df_contract[[f'{contract}_con']]        # first column 
    else:
        df_agg1 = df_agg1.join(df_contract[f'{contract}_con'])     # subsequent columns 

df_agg1

如何在不创建这些不必要的重复项的情况下创建新的数据框? 是什么导致它们被创建?

yingq0072 回答:在创建新的数据框时创建不必要的重复,该新的数据框通过遍历列值从另一个值获取值

这是一种无需使用for循环即可获得完全相同的结果的方法。为了便于阅读,我使用多行代码来添加外植体。

df = pd.DataFrame([["AB1111",'2018-08-15 00:00:00','164'],["AB1111",'2018-08-15 00:03:00','564'],'2018-08-15 00:10:00','364'],'2018-08-15 00:01:00',["CD1111",'2018-08-16 00:12:00','514'],'2018-08-15 00:02:00','2018-08-16 00:05:00','2018-08-16 00:03:00','64'],["EF1111",'534'],'2018-08-17 00:01:00','2018-08-15 00:09:00','524']],columns = ['contract','datetime','real_cons'])


df = df.set_index(['datetime','contract']).unstack().add_suffix('_con')
df = df.droplevel(level=0,axis=1) #drops the 'real_cons' index
df = pd.DataFrame(df.to_records()) #workaround the remove multiindex
df['datetime'] = pd.to_datetime(df['datetime']) #change datetime column to datetime datatype
df = df.set_index('datetime').loc['2018-08-15'] #filter data on date

print(df.reset_index())

结果:

             datetime AB1111_con CD1111_con EF1111_con
0 2018-08-15 00:00:00        164        NaN        534
1 2018-08-15 00:01:00        564        NaN        NaN
2 2018-08-15 00:02:00        NaN        564        NaN
3 2018-08-15 00:03:00        564        NaN        NaN
4 2018-08-15 00:09:00        NaN        NaN        524
5 2018-08-15 00:10:00        364        NaN        NaN
本文链接:https://www.f2er.com/3146956.html

大家都在问