csv.reader do not preserve line return in quoted values

[un-pogaz]

Bug report

As the title say, when reading a CSV file, the line return are not preserved inside a quoted value, even if the reader correctly eager until the end of the quoted value. The line return are just gone.

For the given source file:

"Id (id)","Authors (authors)","Title (title)","Comments (comments)"
"195","Léo Malet","120, Rue de la Gare","Written in 1942.
120, Rue de la Gare was published the following year.

It marked a sort of literary milestone."

The current and simple use of a csv.reader will result to:

[
    ['Id (id)', 'Authors (authors)', 'Title (title)', 'Comments (comments)']
    ['195', 'Léo Malet', '120, Rue de la Gare', 'Written in 1942.120, Rue de la Gare was published the following year.It marked a sort of literary milestone.']
]

expected result:

[
    ['Id (id)', 'Authors (authors)', 'Title (title)', 'Comments (comments)']
    ['195', 'Léo Malet', '120, Rue de la Gare', 'Written in 1942.\n120, Rue de la Gare was published the following year.\n\nIt marked a sort of literary milestone.']
]

Here, as proof of concept, a simple functional python implementation that return the correct values with the preserved line return:

def quoted_line_return(csvfile: Iterable[str]) -> Iterator[str]:
    in_quote = False
    row = []
    for line in csvfile:
        row.append(line)
        if bool(line.count('"') % 2):  # test odd quote in the line
            if in_quote:
                in_quote = False
            else:
                in_quote = True
        if not in_quote:
            yield '\n'.join(row)
            row.clear()

reader = csv.reader(quoted_line_return(lines_src))

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-158837: Clarify csv.reader handling of iterables with stripped newlines. #158909

[utam-1]

@un-pogaz Can I work on this? Though based on what I found out about this, csv.reader just merges together the strings you pass it when it is inside the quotes. If the strings you pass it already got their trailing newlines stripped off, they're lost forever ( for example if you pass it via .splitlines()).
If you pass an iterable that keeps the newlines ( like file obj ), the newlines inside quoted values are correctly preserved.
That's all the more reason to open CSVs with something like this : with open('myFile.csv', newline=' ', encoding='utf-8') as csvfile:, to avoid conflicts with universal file translation.

So, I guess it would be nice to update the documentation so that something like this could be clarified to the users beforehand. I'd open up a PR to do the same should this be desired.

[un-pogaz]

If the strings you pass it already got their trailing newlines stripped off, they're lost forever ( for example if you pass it via .splitlines()).

Oh OOOOH.
Yeah, inded I use a .splitlines(), and after using .splitlines(True) to keep end line return, I have the excepted result.
So, csv.reader work fully as intended, I just passed the wrong input data. Thanks, got it and dully noted.

Consequently, is more a documentation issue that a Python one, and Yes it would be very usefull to update the doc with such warning, because it so easy to misschief with that.

I use .splitlines() precicely to avoid all that os line terminator mess, this is the first time it trick me like that.
btw, ' ' seem to be a illegal value for newline=, you probably want to mean ''.

EDIT: Looking down in the doc, their is kind of a warning for newline sheniganies, but it in on Footnotes. The fact that csv module does its own (universal) newline handling is usefull to know, but IMO it should probably be more prominent.

[utam-1]

I use .splitlines() precicely to avoid all that os line terminator mess, this is the first time it trick me like that. btw, ' ' seem to be a illegal value for newline=, you probably want to mean ''.

EDIT: Looking down in the doc, their is kind of a warning for newline sheniganies, but it in on Footnotes. The fact that csv module does its own (universal) newline handling is usefull to know, but IMO it should probably be more prominent.

Yes, you're correct I made a typo, and I can see it in the Footnote as well, maybe placing it further up along with the example you specified would be better. However, It'd be required that maintainers may approve something like this.