Describe the bug
It seems that DBT core has a dependency on this library, and the latest version 0.5.5 causes some issues with DBT. Specifically larger queries cause the error:
Maximum number of tokens exceeded {number}
I suspect it's this commit:
da67ac1
I realize this is more of a DBT issue than this library, but wanted to give a heads up.
To Reproduce
Using dbt 1.8 or higher, it pulls in this library as dependency.
Not exactly sure, but a fairly large query (sorry I don't have an exact reproducible one I can share)
The limit seemed relatively low.
Expected behavior
Success, these are valid queries that previously had worked.
Aside: terminology of 'tokens' while likely correct gets confusing LLMs being so common these days.
Versions (please complete the following information):
- Python: 3.11.2
- sqlparse: 0.5.5
- dbt: < 1.8 (tested 1.8, 1.9, 1.10)
Additional context
Related DBT issue: dbt-labs/dbt-core#12303
Describe the bug
It seems that DBT core has a dependency on this library, and the latest version 0.5.5 causes some issues with DBT. Specifically larger queries cause the error:
Maximum number of tokens exceeded {number}I suspect it's this commit:
da67ac1
I realize this is more of a DBT issue than this library, but wanted to give a heads up.
To Reproduce
Using dbt 1.8 or higher, it pulls in this library as dependency.
Not exactly sure, but a fairly large query (sorry I don't have an exact reproducible one I can share)
The limit seemed relatively low.
Expected behavior
Success, these are valid queries that previously had worked.
Aside: terminology of 'tokens' while likely correct gets confusing LLMs being so common these days.
Versions (please complete the following information):
Additional context
Related DBT issue: dbt-labs/dbt-core#12303