python – Pandas:使用read_csv时如何获取读取行的状态?

前端之家收集整理的这篇文章主要介绍了python – Pandas:使用read_csv时如何获取读取行的状态?前端之家小编觉得挺不错的,现在分享给大家,也给大家做个参考。

我正在使用pandas和read_csv方法加载一个非常庞大的csv文件,如1000万条记录,我想知道是否有办法显示该加载的进度,如:

  1. 100,000 lines read
  2. 150,000 lines read

谢谢.

最佳答案
显示这样的进度:

  1. Completed 1 %
  2. Completed 2 %
  3. ...
  4. Completed 99 %
  5. Completed 100 %

你可以试试这个:

  1. import os,pandas
  2. filename = "VeryLong.csv"
  3. lines_number = sum(1 for line in open(filename))
  4. lines_in_chunk = 500 # I don't know what size is better
  5. counter = 0
  6. completed = 0
  7. reader = pandas.read_csv(filename,chunksize=lines_in_chunk)
  8. for chunk in reader:
  9. # < ... reading the chunk somehow... >
  10. # showing progress:
  11. counter += lines_in_chunk
  12. new_completed = int(round(float(counter)/lines_number * 100))
  13. if (new_completed > completed):
  14. completed = new_completed
  15. print "Completed",completed,"%"

猜你在找的Python相关文章