Introduce SEA Backend Client with open and close Session functionality #576
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jacky Hu <jacky.hu@databricks.com> Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Co-authored-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
…341) * Redact the URL query parameters from the urllib3.connectionpool logs Signed-off-by: Mubashir Kazia <mubashir.kazia@databricks.com> * Fix code formatting Signed-off-by: Mubashir Kazia <mubashir.kazia@databricks.com> * Add str check for the log record message arg dict values Signed-off-by: Mubashir Kazia <mubashir.kazia@databricks.com> --------- Signed-off-by: Mubashir Kazia <mubashir.kazia@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Jacky Hu <jacky.hu@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* [PECO-1411] Support OAuth InHouse on GCP Signed-off-by: Jacky Hu <jacky.hu@databricks.com> * Update changelog Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> --------- Signed-off-by: Jacky Hu <jacky.hu@databricks.com> Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Co-authored-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* [PECO-1414] Support Databricks InHouse OAuth in Azure Signed-off-by: Jacky Hu <jacky.hu@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Getting ready for test automation Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* update owners Signed-off-by: yunbodeng-db <104732431+yunbodeng-db@users.noreply.github.com> * update owners Signed-off-by: yunbodeng-db <104732431+yunbodeng-db@users.noreply.github.com> * update owners Signed-off-by: yunbodeng-db <104732431+yunbodeng-db@users.noreply.github.com> --------- Signed-off-by: yunbodeng-db <104732431+yunbodeng-db@users.noreply.github.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Jacky Hu <jacky.hu@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* [PECO-1440] Expose current query id on cursor object Signed-off-by: Levko Kravets <levko.ne@gmail.com> * Clear `active_op_handle` when closing the cursor Signed-off-by: Levko Kravets <levko.ne@gmail.com> --------- Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Add a default for retry after Signed-off-by: Ben Cassell <ben.cassell@databricks.com> * Applied black formatter Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Set supports_native_boolean to True Signed-off-by: Alex Holyoke <alexander.holyoke@growthloop.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Don't retry requests that fail with 404 Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev> * Fix lint error Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev> --------- Signed-off-by: Jesse Whitehouse <jesse@whitehouse.dev> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* bump to 3.1.1 Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* fix cookie setting Signed-off-by: Ben Cassell <ben.cassell@databricks.com> * Removing cookie code Signed-off-by: Ben Cassell <ben.cassell@databricks.com> --------- Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Create py.typed Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * add -> Connection annotation Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * massage the code to appease the particular version of the project's mypy deps Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * fix circular import problem Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> --------- Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
fix the return types of the classes' __enter__ functions so that the type information is preserved in context managers eg with-as blocks Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Ben Cassell <ben.cassell@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
changed authentication for proxy Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Relax `pyarrow` pin Signed-off-by: Dave Hirschfeld <dave.hirschfeld@gmail.com> * Allow `pyarrow` 16 Signed-off-by: Dave Hirschfeld <dave.hirschfeld@gmail.com> * Update `poetry.lock` Signed-off-by: Dave Hirschfeld <dave.hirschfeld@gmail.com> --------- Signed-off-by: Dave Hirschfeld <dave.hirschfeld@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Duplicate of applicable change from #93 Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> * Update changelog Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> * Fix after merge Signed-off-by: Levko Kravets <levko.ne@gmail.com> --------- Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Signed-off-by: Levko Kravets <levko.ne@gmail.com> Co-authored-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Enable `delta.feature.allowColumnDefaults` for all tables * Code style Signed-off-by: Levko Kravets <levko.ne@gmail.com> --------- Signed-off-by: Levko Kravets <levko.ne@gmail.com> Co-authored-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: Milan Lukac <milan@lukac.online> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Prepare release 3.2.0 Signed-off-by: Levko Kravets <levko.ne@gmail.com> * Update changelog Signed-off-by: Levko Kravets <levko.ne@gmail.com> --------- Signed-off-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* move py.typed to correct places https://peps.python.org/pep-0561/ says 'For namespace packages (see PEP 420), the py.typed file should be in the submodules of the namespace, to avoid conflicts and for clarity.'. Previously, when I added the py.typed file to this project, #382 , I was unaware this was a namespace package (although, curiously, it seems I had done it right initially and then changed to the wrong way). As PEP 561 warns us, this does create conflicts; other libraries in the databricks namespace package (such as, in my case, databricks-vectorsearch) are then treated as though they are typed, which they are not. This commit moves the py.typed file to the correct places, the submodule folders, fixing that problem. Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * change target of mypy to src/databricks instead of src. I think this might fix the CI code-quality checks failure, but unfortunately I can't replicate that failure locally and the error message is unhelpful Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * Possible workaround for bad error message 'error: --install-types failed (no mypy cache directory)'; see python/mypy#10768 (comment) Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * fix invalid yaml syntax Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * Best fix (#3) Fixes the problem by cding and supplying a flag to mypy (that mypy needs this flag is seemingly fixed/changed in later versions of mypy; but that's another pr altogether...). Also fixes a type error that was somehow in the arguments of the program (?!) (I guess this is because you guys are still using implicit optional) --------- Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * return the old result_links default (#5) Return the old result_links default, make the type optional, & I'm pretty sure the original problem is that add_file_links can't take a None, so these statements should be in the body of the if-statement that ensures it is not None Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * Update src/databricks/sql/utils.py "self.download_manager is unconditionally used later, so must be created. Looks this part of code is totally not covered with tests 🤔" Co-authored-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> --------- Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> Co-authored-by: Levko Kravets <levko.ne@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* Upgrade mypy This commit removes the flag (and cd step) from f53aa37 which we added to get mypy to treat namespaces correctly. This was apparently a bug in mypy, or behavior they decided to change. To get the new behavior, we must upgrade mypy. (This also allows us to remove a couple `# type: ignore` comment that are no longer needed.) This commit runs changes the version of mypy and runs `poetry lock`. It also conforms the whitespace of files in this project to the expectations of various tools and standard (namely: removing trailing whitespace as expected by git and enforcing the existence of one and only one newline at the end of a file as expected by unix and github.) It also uses https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade codebase due to a change in mypy behavior. For a similar reason, it also fixes a new type (or otherwise) errors: * "Return type 'Retry' of 'new' incompatible with return type 'DatabricksRetryPolicy' in supertype 'Retry'" * databricks/sql/auth/retry.py:225: error: object has no attribute update [attr-defined] * /test_param_escaper.py:31: DeprecationWarning: invalid escape sequence \) [as it happens, I think it was also wrong for the string not to be raw, because I'm pretty sure it wants all of its backslashed single-quotes to appear literally with the backslashes, which wasn't happening until now] * ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject [this is like a numpy version thing, which I fixed by being stricter about numpy version] --------- Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> * Incorporate suggestion. I decided the most expedient way of dealing with this type error was just adding the type ignore comment back in, but with a `[attr-defined]` specifier this time. I mean, otherwise I would have to restructure the code or figure out the proper types for a TypedDict for the dict and I don't think that's worth it at the moment. Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> --------- Signed-off-by: wyattscarpenter <wyattscarpenter@gmail.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
earlier, one of the integration tests was failing because 'session was not an attribute of Connection'. This is likely tied to a local configuration issue related to unittest that was causing an error in the test suite itself. The tests are now passing without checking for the session attribute. c676f9b Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
This reverts commit d6b1b19. Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * fixed format Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> * used lazy logging Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> --------- Signed-off-by: Sai Shree Pradhan <saishree.pradhan@databricks.com> Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
…ecouple-session" This reverts commit bdb8381. Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
ensures correctness of self.session.open call in Connection Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
ref: https://github.com/databricks/databricks-jdbc/blob/30688befd95060874b2dd00019dc85a449ed435a/src/main/java/com/databricks/jdbc/common/util/DatabricksThriftUtil.java#L85 Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
Introduce the SEA Backend Client which re-uses the existing Thrift HTTP Client to make HTTP calls to the deployment. Add session opening and closure functionality and related tests.
How is this tested?
Related Tickets & Documents
https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?tab=t.0#heading=h.4cz970y1mk93