>irdZddlmZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZmZddlmZmZmZmZddlmZmZdd lmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ejZe.Z/ee'e(fZ0ee%e&fZ1e r$ddl2m3Z3ddl#m4Z4m5Z5m6Z6ddl7m8Z8ee6e)fZ9ee4e5fZ:GddZ;y)z This module provides a Request class that mimics the behavior of requests. It allows making HTTP requests using the browser's fetch API. ) annotationsN) AsyncIterator)asynccontextmanager) TYPE_CHECKINGAnyCallableOptionalUnioncast)parse_qs urlencodeurlparse urlunparse) HarCapture HarRecorder)Response)RuntimeCommands)Scripts) HTTPError) HeaderEntry) NetworkEventRequestWillBeSentEventRequestWillBeSentExtraInfoEventResponseReceivedEventResponseReceivedExtraInfoEvent$ResponseReceivedExtraInfoEventParams) CookieParam ResourceType)Tab)RequestWillBeSentEventParams%RequestWillBeSentExtraInfoEventParamsResponseReceivedEventParams)EvaluateResponsecdeZdZdZd$dZ d% d&dZ d' d(dZ d) d*dZ d) d*dZ d) d*dZ d+d Z d+d Z d+d Z e d' d,d Zed-d Z d.dZ d/dZed0dZe d1dZd2dZe d3dZd4dZd4dZd5dZd5dZe d6dZ d7dZ d8dZ d9dZ d:dZed;dZ dd Z#ed?d!Z$ed@d"Z%edAd#Z&y)BRequesta High-level interface for making HTTP requests using the browser's fetch API. This class provides a requests-like interface that executes HTTP requests in the browser's JavaScript context. All requests inherit the browser's current session state including cookies, authentication headers, and other automatic browser behaviors. This allows for seamless interaction with websites that require authentication or have complex cookie management. Key Features: - Executes requests in the browser's JavaScript context using fetch API - Automatically includes browser cookies and session state - Preserves browser's security context and CORS policies - Captures both request and response headers for analysis - Supports all standard HTTP methods (GET, POST, PUT, DELETE, etc.) Note: - Headers passed to methods are additional headers, not replacements - Browser's automatic headers (User-Agent, Accept, etc.) are preserved - Cookies are managed automatically by the browser ct||_d|_g|_g|_g|_t j dy)a:Initialize a new Request instance bound to a browser tab. Args: tab: The browser tab instance where requests will be executed. This tab provides the JavaScript execution context and maintains the browser's session state (cookies, authentication, etc.). Fz"Request helper initialized for tabN)tab_network_events_enabled _callback_ids_requests_sent_requests_receivedloggerdebug)selfr's e/var/www/python/tenders/Scrapper/venv/lib/python3.12/site-packages/pydoll/browser/requests/request.py__init__zRequest.__init__Ss8',$(*68>@ 9:Nc K|j||}|j||||fi|} tjd|j d|tj d|j d|dt |d|dud|du  |j|| d{} |j} |j} |j} |j| | | | |jd{S7`7#t$r5}tjd|tdt!||d}~wwxYw#|jd{7wxYww)aExecute an HTTP request in the browser's JavaScript context. This method uses the browser's fetch API to make requests, inheriting all browser session state including cookies, authentication, and security context. The request is executed as if made by the browser itself. Args: method: HTTP method (GET, POST, PUT, DELETE, etc.). Case insensitive. url: Target URL for the request. Can be relative or absolute. params: Query parameters to append to the URL. These are URL-encoded and merged with any existing query string in the URL. data: Request body data. Behavior depends on type: - dict/list/tuple: URL-encoded as form data (application/x-www-form-urlencoded) - str/bytes: Sent as-is with no Content-Type modification Mutually exclusive with 'json' parameter. json: Data to be JSON-serialized as request body. Automatically sets Content-Type to application/json. Mutually exclusive with 'data'. headers: Additional headers to include. These are ADDED to browser's automatic headers, not replacements. Format: [{'name': 'X-Custom', 'value': 'value'}] **kwargs: Additional fetch API options (e.g., credentials, mode, cache). Returns: Response object containing status, headers, content, and cookies from both the request and response phases. Raises: HTTPError: If the request execution fails or network error occurs. Note: - Browser cookies are automatically included - CORS policies are enforced by the browser - Authentication headers are preserved from browser session zExecuting request: method=z, url=z , headers=z, json=Nz, data=zRequest failed: )_build_url_with_params_build_request_optionsr,infoupperr-bool_execute_fetch_request_extract_received_headers_extract_sent_headers_extract_set_cookies_build_response_clear_callbacks Exceptionerrorrstr)r.methodurlparamsdatajsonheaderskwargs final_urloptionsresultreceived_headers sent_headerscookiesexcs r/requestzRequest.requestbswX//V< -$--fgtTTVT 00@ykRS ((8ykJG}oWT-=,>gdRVFVEW Y  *66y'JJF#==? 557L//1G''0@,PWX'') ) )K *  D LL+C51 2.s3xj9: C D '') ) )sgBE-D)D *AD1E-D E- D E- E 0EE  EE*#E&$E**E-cJK|jd|fd|i|d{S7w)aExecute a GET request for retrieving data. Args: url: Target URL to retrieve data from. params: Query parameters to append to URL. **kwargs: Additional fetch options. Returns: Response object with retrieved data. GETrCNrO)r.rBrCrGs r/getz Request.gets+ "T\\%FVFvFFFFs #!#cLK|jd|f||d|d{S7w)adExecute a POST request for creating or submitting data. Args: url: Target URL for data submission. data: Form data to submit (URL-encoded). json: JSON data to submit. **kwargs: Additional fetch options. Returns: Response object with server's response to the submission. POSTrDrENrRr.rBrDrErGs r/postz Request.posts-$"T\\&#NDtNvNNNN $"$cLK|jd|f||d|d{S7w)aYExecute a PUT request for updating/replacing resources. Args: url: Target URL of resource to update. data: Form data for the update. json: JSON data for the update. **kwargs: Additional fetch options. Returns: Response object confirming the update operation. PUTrVNrRrWs r/putz Request.puts-$"T\\%M4dMfMMMMrYcLK|jd|f||d|d{S7w)amExecute a PATCH request for partial resource updates. Args: url: Target URL of resource to partially update. data: Form data with changes to apply. json: JSON data with changes to apply. **kwargs: Additional fetch options. Returns: Response object confirming the partial update. PATCHrVNrRrWs r/patchz Request.patchs-$"T\\'3OTOOOOOrYcFK|jd|fi|d{S7w)zExecute a DELETE request for removing resources. Args: url: Target URL of resource to delete. **kwargs: Additional fetch options. Returns: Response object confirming the deletion. DELETENrRr.rBrGs r/deletezRequest.deletes&"T\\(C:6:::: !!cFK|jd|fi|d{S7w)a}Execute a HEAD request to retrieve only response headers. Useful for checking resource existence, size, or modification date without downloading the full content. Args: url: Target URL to check headers for. **kwargs: Additional fetch options. Returns: Response object with headers but no body content. HEADNrRrbs r/headz Request.heads&"T\\&#88888rdcFK|jd|fi|d{S7w)aaExecute an OPTIONS request to check allowed methods and capabilities. Used for CORS preflight checks and discovering server capabilities. Args: url: Target URL to check options for. **kwargs: Additional fetch options. Returns: Response object with allowed methods and CORS headers. OPTIONSNrRrbs r/rIzRequest.options s&"T\\)S;F;;;;rdcKt|j|}t|}|jd{ ||j d{y7#7#|j d{7wxYww)aRecord network traffic as HAR. Context manager that captures all network activity on the tab and produces a HarCapture object for export. Args: resource_types: Optional list of resource types to capture. When provided, only requests matching these types are recorded. When None (default), all resource types are captured. Usage:: async with tab.request.record() as capture: await tab.go_to('https://example.com') capture.save('flow.har') # Record only fetch and XHR requests async with tab.request.record( resource_types=[ResourceType.FETCH, ResourceType.XHR] ) as capture: await tab.go_to('https://example.com') capture.save('api_calls.har') Yields: HarCapture: Object with .save(), .to_dict(), and .entries. )resource_typesN)rr'rstartstop)r.rkrecordercaptures r/recordzRequest.recordsf@txxGX&nn "M--/ ! !  "(--/ ! !sD6A>AA>A A>AA>A> A;4A75A;;A>ctjd|d||s|St|}t|j}|j D] \}}|g||< t |jt|dS)z&Build final URL with query parameters.zBuilding URL with params: url=z , params=Tdoseq)query) r,r-rr rtitemsr_replacer )rBrCparsedrtkeyvalues r/r3zRequest._build_url_with_paramsBs  5cU)F8LMJ#& ,,. !JCE#J !&// %t0L/MNNr1c |r|j|ni}|j|d|}tjd||j ||||S)z!Build request options dictionary.)rArFz"Building request options: options=)_convert_header_entries_to_dictr6r,r-_add_request_body)r.rArFrErDrG headers_dictrIs r/r4zRequest._build_request_optionsPseIPt;;GDUW lln#    9'CD wd3r1cX||j||y||j||yy)z5Add request body and appropriate Content-Type header.N)_handle_json_options_handle_data_options)r.rIrErDs r/r|zRequest._add_request_bodycs5    % %gt 4    % %gt 4r1ctj||d<|djddtj dy)zHandle JSON options.bodyrF Content-Typezapplication/jsonz.Request JSON body set and content-type appliedN)jsonlibdumps setdefaultr,r-)rIrEs r/rzRequest._handle_json_optionsos9"--- %%n6HI EFr1ct|tttfr;t |d|d<|dj ddt jdy ||d<t jdy ) zHandle data options.TrrrrFrz!application/x-www-form-urlencodedz'Request data encoded as form-urlencodedzRequest data set as raw payloadN) isinstancedictlisttupler rr,r-)rIrDs r/rzRequest._handle_data_optionsvs^ dT4/ 0'D9GFO I  ) ).:] ^ LLB C"GFO LL: ;r1clKtjjtj|tj|}|j d{t jd|jjtj|ddd{S7R7w)z2Execute the fetch request using browser's runtime.)rBrINzERegistered network callbacks and executing fetch via Runtime.evaluateT) expressionreturn_by_value await_promise) r MAKE_REQUESTformatrr_register_callbacksr,r-r'_execute_commandrevaluate)r.rBrIscripts r/r8zRequest._execute_fetch_requests%%,,s1CW]][bMc,d&&((( \]XX..  $ $! $"     ) s%AB4B0A B4+B2,B42B4c |ddd}tjd|t|dt|j dd|d|d||||d  S) z(Build Response object from fetch result.rJryz Building response: result_value=statuscontentr1textrErB) status_coderrrEresponse_headersrequest_headersrMrB)r,r-rbytesrS)rJrrrM result_values r/r<zRequest._build_responseswh'1':  7 ~FG$X.,**9c:;f%f%-+U#  r1cKjjs>jjd{d_tj ddfd }dfd }jj tj|d{jj tj|d{jj tj|d{jj tj|d{g_ y777s7C7w)aRegister network event listeners to capture request/response metadata. Sets up CDP event listeners to capture all network activity during the request execution. This includes both outgoing request data and incoming response data, which are used for header and cookie extraction. Note: Network events are only enabled if not already active on the tab. NTz1Network events enabled on tab for request capturecjjtt|tj d|y)Nz!Appended received request: event=)r+appendr RequestReceivedEventr,r-eventr.s r/append_received_requestz.append_received_requests3  # # * *40De+L M LL.append_sent_requests3    & &t,D4AD>D61D>D81D>6D:71D>(D< ) D>6D>8D>:D><D>cFK|jD]%}|jj|d{'|jj|jr?|jj d{d|_t jdyy7l7$w)zClean up network event listeners and disable network monitoring. Removes only the callbacks registered by this request instance (surgical removal) so other listeners (e.g. HarRecorder) are not affected. NFz,Network events disabled on tab after request)r)r'remove_callbackclearr(disable_network_eventsr,r-)r. callback_ids r/r=zRequest._clear_callbackss -- 8K((**;7 7 7 8   "  ' '((113 3 3+0D ( LLG H ( 8 4s".B!BA B!:B;#B!B!cl|j|jd}|j|j|S)zExtract headers from response network events. Returns: List of headers received from the server during response. )responseblockedCookies)"_extract_response_received_headers-_extract_response_received_extra_info_headers_extract_headers_from_eventsr+r.event_extractorss r/r9z!Request._extract_received_headerss;??"PPK  001H1HJZ[[r1cl|j|jd}|j|j|S)zExtract headers from request network events. Returns: List of headers that were actually sent in the request. )rOassociatedCookies)_extract_request_sent_headers(_extract_request_sent_extra_info_headersrr*rs r/r:zRequest._extract_sent_headerss;99!%!N!NK  001D1DFVWWr1cg}t}tjd||D]}|d}|jD]\}}||vs ||}tjd||D]c} | d| df} tjd| | |vs*|j | |j | tjd| etjd||S) aExtract headers from network events using appropriate extractors. Args: events: List of network events to process. event_extractors: Mapping of event keys to header extraction functions. Returns: Deduplicated list of headers from all matching events. Note: Headers are deduplicated based on name-value pairs to avoid duplicate entries from multiple event types. z'Extracting headers from events: events=rCz%Extracted headers: extracted_headers=nameryzIdentity: identity=zAdded header: header=zHeaders extracted: headers=)setr,r-ruradd) eventsrrFseenrrCrx extractorextracted_headersheaderidentitys r/rz$Request._extract_headers_from_eventss $&(u >vhGH E8_F"2"8"8": Y&=(1&(9%LL#HIZH[!\]"3K$*6NF7O#D ':8*%EF#4/#NN62 HHX."LL+@)IJ K    27)<=r1c~|d}tjd||j|jdiS)zExtract headers from main request event. Args: params: Event parameters containing request details. Returns: List of headers that were sent with the request. rOz)Extracting request sent headers: request=rFr,r-_convert_dict_to_header_entriesrS)r.rCrOs r/rz%Request._extract_request_sent_headers s># @ JK33GKK 24NOOr1cttjd||j|jdiS)a}Extract headers from extra request info event. This event contains additional header information that may not be present in the main request event, such as security-related headers. Args: params: Extra info event parameters containing additional headers. Returns: List of additional headers sent with the request. z3Extracting request sent extra info headers: params=rFrr.rCs r/rz0Request._extract_request_sent_extra_info_headers/s4  J6(ST33FJJy"4MNNr1c~|d}tjd||j|jdiS)zExtract headers from main response event. Args: params: Event parameters containing response details. Returns: List of headers received from the server. rz/Extracting response received headers: response=rFr)r.rCrs r/rz*Request._extract_response_received_headers@s>*% FxjQR33HLLB4OPPr1cttjd||j|jdiS)aExtract headers from extra response info event. This event contains additional response header information, including Set-Cookie headers and security-related headers that may be filtered from the main response event. Args: params: Extra info event parameters containing additional headers. Returns: List of additional headers received from the server. z8Extracting response received extra info headers: params=rFrrs r/rz5Request._extract_response_received_extra_info_headersOs5  OPVxXY33FJJy"4MNNr1ctjd||jDcgc]\}}t||c}}Scc}}w)zConvert header dictionary to standardized HeaderEntry format. Args: headers_dict: Dictionary mapping header names to values. Returns: List of HeaderEntry objects with 'name' and 'value' keys. z6Converting dictionary to header entries: headers_dict=rry)r,r-rur)r}rrys r/rz'Request._convert_dict_to_header_entriesas@  Ml^\]GSGYGYG[\ e U3\\\sAc g}tjd||j}tjd||D]}tt|d}|j |d}tjd||Dcgc]}|ddk(s |d}}tjd ||sx|D]#}|j ||j|%tjd ||Scc}w) agExtract and parse all Set-Cookie headers from response events. Processes response events to find Set-Cookie headers and converts them into structured cookie objects. Handles multiple Set-Cookie headers and multi-line cookie declarations. Returns: List of unique cookies extracted from Set-Cookie headers. z Extracting set cookies: cookies=zAFiltering response extra info events: response_extra_info_events=rCrFz1Converting dictionary to header entries: headers=rz Set-Cookieryz'Set cookie headers: set_cookie_headers=zSet cookies extracted: cookies=)r,r-"_filter_response_extra_info_eventsr rr_add_unique_cookies_parse_set_cookie_header) r.rMresponse_extra_info_eventsrrCrFrset_cookie_headersset_cookie_headers r/r;zRequest._extract_set_cookiesns(&( 7yAB%)%L%L%N" **D)E G 0 E>hPF::6);LMG LLLWIV W.5"$*<9Ww" " LLBCUBVW X!);%,,!>!>?P!Q   6wi@A"s DDctjd|j|jDcgc]}|dtjk(r|c}Scc}w)zFilter network events to find those containing Set-Cookie information. Returns: List of events that contain extra response information including cookies. z8Filtering response extra info events: requests_received=rA)r,r-r+rr)r.rs r/rz*Request._filter_response_extra_info_eventss[  FtG^G^F_ ` 00 X,"K"KK    sAcg}|jd}tjd||D]?}|j|}|stjd||j |Atjd||S)aParse a Set-Cookie header value into individual cookie objects. Handles both single and multi-line Set-Cookie headers, extracting cookie name-value pairs while ignoring attributes like Path, Domain, etc. Args: set_cookie_header: Raw Set-Cookie header value from HTTP response. Returns: List of parsed cookie objects with name and value.  z-Parsing set cookie header: set_cookie_header=zParsed cookie: cookie=zParsed cookies: cookies=)splitr,r-_parse_cookie_liner)r.rrMlineslinecookies r/rz Request._parse_set_cookie_headers!''- DEVDWXY 'D,,T2F 5fX>?v&  '  /y9:r1cd|vry|jdddj}|jdddjdddj}|syt||S)aParse a single cookie line to extract name and value. Extracts only the cookie name and value, ignoring all cookie attributes like Path, Domain, Secure, HttpOnly, etc. Rejects cookies with empty names. Args: line: Single line from Set-Cookie header. Returns: CookieParam object with name and value, or None if parsing fails or name is empty. =Nr;r)rstripr)rrrys r/rzRequest._parse_cookie_linesq d?zz#q!!$**, 3"1%++C3A6<<>E22r1ctjd|d||D]0}||vs|j|tjd|2tjd|y)zAdd cookies to list while avoiding duplicates. Args: cookies: Existing list of cookies to add to. new_cookies: New cookies to add if not already present. zAdding unique cookies: cookies=z, new_cookies=zAdded unique cookie: cookie=zUnique cookies added: cookies=N)r,r-r)rM new_cookiesrs r/rzRequest._add_unique_cookiessm  6wi~k][\! FFW$v& ;F8DE F  5gY?@r1cjtjd||Dcic] }|d|d c}Scc}w)apConvert HeaderEntry objects to a plain dictionary format. Used for preparing headers for the JavaScript fetch API which expects a simple object mapping header names to values. Args: headers: List of HeaderEntry objects with 'name' and 'value' keys. Returns: Dictionary mapping header names to values. z1Converting header entries to dictionary: headers=rry)r,r-)rFrs r/r{z'Request._convert_header_entries_to_dicts9  H RS>EFFvw/FFFs0)r'r)NNNN)rAr@rBr@rCOptional[dict[str, str]]rD.Optional[Union[dict, list, tuple, str, bytes]]rEOptional[dict[str, Any]]rFOptional[list[HeaderEntry]]rr)N)rBr@rCrrr)NN)rBr@rDrrErrr)rBr@rr)rkzlist[ResourceType] | NonerzAsyncIterator[HarCapture])rBr@rCrrr@) rAr@rFrrErrDrrdict[str, Any])rIrrErrDrrr)rIrrErrr)rIrrDrrr)rBr@rIrrr#) rJr#rlist[HeaderEntry]rrrMlist[CookieParam]rr)rr)rr)rz9Union[list[RequestSentEvent], list[RequestReceivedEvent]]rz-dict[str, Callable[[Any], list[HeaderEntry]]]rr)rCr rr)rCr!rr)rCr"rr)rCrrr)r}rrr)rr)rzlist[RequestReceivedEvent])rr@rr)rr@rzOptional[CookieParam])rMrrrrr)rFrrzdict[str, str])'__name__ __module__ __qualname____doc__r0rOrSrXr\r_rcrgrIrrp staticmethodr3r4r|rrr8r<rr=r9r:rrrrrrr;rrrrr{r1r/r%r%=s* ;&,0?C)-/3?*?*?*) ?* = ?* ' ?*-?* ?*H,0G G)G  G*@D)- O O=O' O  O.@D)- N N=N' N  N.@D)- P P=P' P  P( ; 9 <59%"1%" #%"%"N O O-'  =  & 5 5' 5= 5  5GG  < <'U <  < <     + + #     (( T I \ X$I$G$ $$L P2 P  PO;O O" Q1 Q  QO:O O$ ] ]B  .330 A A G Gr1r%)rs #)*FFBBH5<$'3D   8 $"$#% & A!&#, ."#$- / n Gn Gr1