У меня есть CSV-файл с 100 000 строк, и я хочу читать 10 строк за раз и обрабатывать каждую строку, чтобы каждый раз сохранять в соответствующий файл и спать в течение 5 секунд. Я пытаюсь Nslice, но он читает только первые 10 и останавливается. Я хочу, чтобы программа работала до EOF. Я использую jupyter, python2 и pyspark, если это поможет.

from itertools import islice
with open("per-vehicle-records-2020-01-31.csv") as f:
    while True:
        next_n_lines = list(islice(f, 10))
        if not next_n_lines:
            break
        else:
            print(next_n_lines)
            sleep(5)

Это не разделяет каждый ряд. Объединяет 10 строк в список

['"cosit","year","month","day","hour","minute","second","millisecond","minuteofday","lane","lanename","straddlelane","straddlelanename","class","classname","length","headway","gap","speed","weight","temperature","duration","validitycode","numberofaxles","axleweights","axlespacings"\n', '"000000000997","2020","1","31","1","30","2","0","90","1","Test1","0","","5","HGV_RIG","11.4","2.88","3.24","70.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","3","0","90","2","Test2","0","","2","CAR","5.2","3.17","2.92","71.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","5","0","90","1","Test1","0","","2","CAR","5.1","2.85","2.51","70.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","6","0","90","2","Test2","0","","2","CAR","5.1","3.0","2.94","69.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","9","0","90","1","Test1","0","","5","HGV_RIG","11.5","3.45","3.74","70.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","10","0","90","2","Test2","0","","2","CAR","5.4","3.32","3.43","71.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","13","0","90","2","Test2","0","","2","CAR","5.3","3.19","3.23","71.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","13","0","90","1","Test1","0","","2","CAR","5.2","3.45","3.21","70.0","0.0","0.0","0","0","0","",""\n', '"000000000997","2020","1","31","1","30","16","0","90","1","Test1","0","","5","HGV_RIG","11.0","2.9","3.13","69.0","0.0","0.0","0","0","0","",""\n']
0
Kush Singh 26 Апр 2020 в 14:35

2 ответа

Лучший ответ

Это должно работать:

import pandas as pd
import time
path_data = 'per-vehicle-records-2020-01-31.csv'

reader = pd.read_csv(path_data, sep=';', chunksize=10, iterator=True)
for i in reader:
    df = next(reader)
    print(df)
    time.sleep(5) 

Размер фрагмента будет читать каждые 10 строк, и цикл for должен следить за тем, чтобы они читались таким образом, и спать 5 секунд между каждой итерацией.

0
Naman Doctor 26 Апр 2020 в 12:11

islice повторно запустит генератор, поэтому вам нужно повторить после назначения Это

from itertools import islice
with open("per-vehicle-records-2020-01-31.csv") as f:
    while True:
        next_n_lines = islice(f, 10)
        if not next_n_lines:
            break
        else:
            for line in next_n_lines:
               print(line)
            sleep(5)

Вы читаете больше здесь Как прочитать файл N строк в то время в Python?

0
Beny Gj 26 Апр 2020 в 12:01