wrong parsing

[piv199]

"They used to work here, _____ they?",hadn't,weren't,didn't,used not,0,0,1,0,4

Created through Microsoft Excel, but "They used to work here, _____ they?" parsed as two different tokens, instead of one.

So quotes are ignored, it seems...

[Daniel1of1]

Thanks for the issue. For reference - what is the your expected outcome for the above string?

I assume:

[
"They used to work here, _____ they?",
"hadn't",
"weren't",
"didn't",
"used not",
"0",
"0",
"1",
"0",
"4"
]

Is that correct?

[piv199]

@Daniel1of1 yes, so if the string starts with " then the whole token is up to ",, as far as I understand from excel.

Also there could be " inside token, they should be left as is.

[Daniel1of1]

Incase this is still an issue for you, the latest develop should have this behaviour now.
And to make clear my previous point expected string should actually be: (different first line)

[
"\"They used to work here, _____ they?\"",
"hadn't",
"weren't",
"didn't",
"used not",
"0",
"0",
"1",
"0",
"4"
]

Right?

( no worries if this long past being an issue 😊 )